Outage in Grafana

Grafana Cloud: Prometheus increased series generation due to Tempo update

Resolved Minor
May 22, 2023 - Started over 2 years ago - Lasted 1 day
Official incident page

Need to monitor Grafana outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Grafana, and never miss an outage again.
Start Free Trial

Outage Details

During a weekly update cycle, we implemented a modification to the Tempo system that inadvertently led to a significant increase in the number of series being produced by the Metrics Generator. This issue affected multiple regions for a duration of 6-11 hours. We have since identified the root cause and have deployed a fix to address it.
Components affected
Grafana AZURE - Ingestion Grafana AZURE - Querying Grafana US-CENTRAL: Querying Grafana US-CENTRAL: Ingestion Grafana EU-WEST: Ingestion Grafana EU-WEST: Querying
Latest Updates ( sorted recent to last )
RESOLVED over 2 years ago - at 05/23/2023 08:51AM

This incident has been resolved.

MONITORING over 2 years ago - at 05/22/2023 09:42PM

This issue has been resolved and we are currently monitoring.

MONITORING over 2 years ago - at 05/22/2023 09:42PM

During a weekly update cycle, we implemented a modification to the Tempo system that inadvertently led to a significant increase in the number of series being produced by the Metrics Generator. This issue affected multiple regions for a duration of 6-11 hours. We have since identified the root cause and have deployed a fix to address it.

Latest Grafana outages

Google Cloud Monitoring data source degraded performance in prod-us-central-0 - about 11 hours ago
Hosted Grafana - Grafana instances crashlooping - about 17 hours ago
Failures to ingest otel metrics .net apps - 7 days ago
Synthetic monitoring API outage - us-central2 - 8 days ago
hgapi down in prod-us-west-0 - 9 days ago

Be the First to Know When Vendors Go Down

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4484 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook