Grafana Outage History

Grafana Cloud Metrics in the prod-us-central-7 region experienced high write latency and errors affecting data ingestion and rule evaluation from approximately 20:40-21:00 UTC. Users encountered slow performance and errors when writing metrics data during this 20-minute window. The team identified the root cause, implemented a fix, and confirmed stability after monitoring the system for an additional 1.5 hours.

Grafana experienced metrics read errors in the prod-ap-south-1 region starting at 06:24 UTC, causing users to encounter error messages when querying metrics. The issue affected the AWS India region's querying functionality for 41 minutes. Engineering released a fix at 07:50 UTC, resolving the query errors and restoring normal service.

Grafana experienced datasource query performance issues in the prod-us-east-4 region, affecting Grafana Cloud Integrations. The incident lasted 6 minutes and was classified as minor. The team investigated and resolved the performance degradation that was impacting query response times.

Grafana experienced elevated error rates affecting browser checks in the PoP Oregon region, impacting the AWS US East prod-us-east-0 Public Probes component. The team identified the root cause and implemented a fix after 4.1 hours, with services recovering during the monitoring phase before full resolution was confirmed.

Grafana experienced a major outage affecting k6 and Synthetic Monitoring services across all global regions, impacting APIs and public probes on AWS, Azure, and GCP platforms. The incident lasted 3.3 hours before a fix was implemented and services were restored. The team successfully resolved the issue after identifying the root cause and monitoring the recovery.

Grafana experienced ingestion errors for AWS Cloud Provider Observability Metric Streams in the prod-us-central-7 region starting around 06:30 UTC. Users in this region encountered metric ingestion failures when using AWS Metric Streams. The issue was resolved after 1.3 hours with a fix implemented by engineering.

Grafana experienced an 18.9-hour incident where users were unable to save credentials for SQL-based data sources, affecting a subset of customers across multiple regions. The issue impacted Grafana Cloud Integrations functionality, preventing proper configuration of SQL datasource connections. The incident was resolved after the team identified the root cause and implemented a fix.

Grafana experienced gateway slowness in the US-East-1 production environment, with dropped successful requests potentially preventing users from accessing their instances. The incident lasted 53.9 hours but was ultimately determined to be a false alarm that did not actually affect any users.

The Grafana InfluxDB datasource plugin experienced intermittent failures affecting Grafana Cloud Integrations for 6.3 hours. Users encountered connection issues when trying to access data through the InfluxDB plugin. The team identified the root cause, implemented a fix, and fully resolved the incident after monitoring to confirm service recovery.

Grafana experienced a major outage affecting Cloudwatch datasources in the GCP US Central region for 5.6 hours. Users were unable to access or query data from their Cloudwatch datasources during this period. The issue was resolved after the team implemented a fix and confirmed full service recovery.