Outage in Grafana

Grafana Cloud: Prometheus increased series generation due to Tempo update

Resolved Minor
May 22, 2023 - Started over 2 years ago - Lasted 1 day
Official incident page

Incident Report

During a weekly update cycle, we implemented a modification to the Tempo system that inadvertently led to a significant increase in the number of series being produced by the Metrics Generator. This issue affected multiple regions for a duration of 6-11 hours. We have since identified the root cause and have deployed a fix to address it.
Components affected
Grafana AZURE - Ingestion Grafana AZURE - Querying Grafana US-CENTRAL: Querying Grafana US-CENTRAL: Ingestion Grafana EU-WEST: Ingestion Grafana EU-WEST: Querying
The Status Page Aggregator Built for IT Teams

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Try IsDown risk-free 14-day free trial · No credit card required
Latest Updates ( sorted recent to last )
RESOLVED over 2 years ago - at 05/23/2023 08:51AM

This incident has been resolved.

MONITORING over 2 years ago - at 05/22/2023 09:42PM

This issue has been resolved and we are currently monitoring.

MONITORING over 2 years ago - at 05/22/2023 09:42PM

During a weekly update cycle, we implemented a modification to the Tempo system that inadvertently led to a significant increase in the number of series being produced by the Metrics Generator. This issue affected multiple regions for a duration of 6-11 hours. We have since identified the root cause and have deployed a fix to address it.

Latest Grafana outages

New users registered in Synthetic Monitoring from October 15th are no able to create checks - about 23 hours ago
Synthetic Probe Connectivity Issues - 2 days ago
Loki degraded writes in SouthEast region - 3 days ago
Machine learning unavailable for the prod-us-west-0 region - 6 days ago
AWS Firehose Logs Down in Prod-Us-West-0 - 7 days ago

The Status Page Aggregator Built for IT Teams

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4522 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook