Grafana Outage History

Grafana experienced degraded query caching performance across multiple regions (EU-WEST, US-EAST, and US-CENTRAL) starting at 20:52 UTC, causing queries to take longer than usual for datasources with query caching configured. The incident affected production environments in Azure Netherlands, AWS US East, and GCP US Central regions. The issue was resolved after 1.6 hours, with recovery occurring in stages across the affected regions.

Grafana experienced stack creation failures across all regions starting at 12:11 UTC on April 16th, with customers receiving error messages when attempting to create new stacks. The issue was identified as originating from an external provider rather than Grafana's own systems. The incident was resolved after 1.2 hours of downtime, with service restored and monitored before being declared fully resolved.

The Grafana support team experienced a 20-minute issue with their Zendesk ticketing system where tickets were not displaying properly in internal support views, though all new tickets continued to be received without loss. The team actively monitored incoming requests and worked around the visibility issue to ensure all tickets were still reviewed. The incident was resolved and the ticketing system returned to full operational status.

Grafana's K6 service experienced sporadic DNS issues from April 9-15 that caused cloud test runs to occasionally fail to start and abort. The problem was traced to a flaky DNS server that was causing random test initialization failures. The engineering team deployed a fix to resolve the DNS server issues and confirmed full resolution after monitoring the system.

Grafana Cloud Logs experienced write path degradation in the us-east-3 cluster, preventing log ingestion through Loki. The incident lasted 46 minutes before a fix was implemented and the service was restored to normal operation.

Grafana's Tempo service experienced a write outage in the prod-us-east-3 region starting at 18:50 UTC, causing users to encounter errors, timeouts, and service unavailability. The engineering team implemented a fix and monitored the recovery process. The incident was fully resolved after 1.4 hours of downtime.

Grafana's K6 browser testing service experienced an issue where users running browser tests could not view the browser timeline. The incident affected the visibility of browser test timelines, preventing users from accessing this diagnostic information. The team identified the root cause and implemented a fix, resolving the issue after 1.3 hours.

Grafana experienced a 5-hour incident where users were unable to edit notification policies across all Alertmanager components in multiple regions worldwide. The team identified the root cause and implemented a fix to restore full functionality. The incident has been resolved with notification policy editing capabilities restored globally.

Grafana experienced a UI issue where notification policies and contact points were missing from the interface for instances on the slow release channel, while the underlying API calls continued to function normally. The incident affected Alertmanager and Rule Evaluation components across multiple regions globally for approximately 21.7 hours. The team identified the root cause, implemented a fix, and monitored the recovery before confirming full resolution.

Grafana experienced a 7-minute outage that prevented users from executing k6 test runs that use extensions, affecting both local and Grafana Cloud environments. Test runs without extensions continued to function normally during the incident. The issue was resolved after 7 minutes of investigation.