Grafana Outage History

Grafana's AWS integration experienced degraded performance starting around 18:15 UTC, causing scrapes to hit rate limits and resulting in missing data points for serverless integrations. The issue intermittently affected all customers using the AWS integration across all regions. The incident was resolved after 48 minutes.

Grafana's prod-eu-west-0.cortex-prod-01 metrics cell experienced data ingestion delays that caused partial query results and failed rule evaluations, affecting querying and ingestion services in the EU-WEST region and GCP Belgium. The incident lasted 11.3 hours before being resolved with an implemented fix. Users in the affected regions would have experienced incomplete or failed metric queries and alerting rule failures during this period.

Grafana's Synthetic Monitoring service experienced a major incident where scripted and browser check creation and updates were blocked through the plugin app across multiple global probe locations. The issue was isolated to the plugin app interface only, with Terraform and direct API access remaining functional. The incident was resolved after 28 minutes once the team identified the root cause and implemented a fix.

Grafana experienced a major incident where CloudWatch queries were failing across multiple regions including US, EU, Asia-Pacific, and Brazil deployments. The issue affected 26 components globally, impacting users' ability to retrieve monitoring data from AWS CloudWatch. The incident was resolved after 37 minutes of downtime, with monitoring continuing to ensure stability.

Grafana experienced a major outage lasting 7.2 hours that primarily affected Free tier users across all global regions, with impacted users encountering an indefinite "your Grafana instance is loading" message when trying to access their instances. The incident affected numerous AWS, Azure, and GCP regions worldwide, as well as the play.grafana.org service. The team identified the root cause after several hours of investigation, implemented a fix, and fully resolved the issue.

Grafana experienced degraded Prometheus writes in the prod-eu-west-3 region starting at 08:45Z, which later expanded to impact Logs and Synthetic Monitoring services. The incident affected ingestion, API, and public probes, causing errors in check execution metrics, potential missed alerts for Synthetic Monitoring, and gaps in recording rules for Logs due to delayed remote writes to Mimir. The issue was resolved after 27.7 hours with a fix implemented by the engineering team.

Grafana Assistant was completely unavailable in the prod-us-east-0 region for 1.8 hours, with users encountering a Terms of Service acceptance prompt that failed to function properly. The engineering team identified the root cause and implemented a fix to restore full service functionality.

Grafana's Authentication API database experienced failures in the prod-eu-west-2 and prod-eu-west-4 regions, with database writes failing while reads remained operational. The incident affected AWS Germany regions and impacted users' ability to perform authentication-related operations that required database writes. The issue was resolved after 40 minutes of downtime.

Grafana Cloud experienced a major incident affecting multiple datasources including CloudWatch, Aurora, Opensearch, X-Ray, Timestream, Redshift, and Sitewise, causing failures in the Integrations component. The issue was identified and a fix was rolled out, with recovery observed for affected datasources. The incident was resolved after 2 hours of monitoring and remediation efforts.

Grafana Cloud k6 experienced degraded performance and errors affecting certain v6 API endpoints for 6.9 hours. Customers encountered performance issues and errors when using these endpoints during the incident. The service team investigated and resolved the API endpoint problems.