Outage in Grafana

Grafana Cloud: Prometheus increased series generation due to Tempo update

Resolved Minor
May 22, 2023 - Started 12 months ago - Lasted 1 day
Official incident page

Need to monitor Grafana outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Grafana, and never miss an outage again.
Start Free Trial

Outage Details

During a weekly update cycle, we implemented a modification to the Tempo system that inadvertently led to a significant increase in the number of series being produced by the Metrics Generator. This issue affected multiple regions for a duration of 6-11 hours. We have since identified the root cause and have deployed a fix to address it.
Components affected
Grafana AZURE - Ingestion Grafana AZURE - Querying Grafana US-CENTRAL: Querying Grafana US-CENTRAL: Ingestion Grafana EU-WEST: Ingestion Grafana EU-WEST: Querying
Latest Updates ( sorted recent to last )
RESOLVED 12 months ago - at 05/23/2023 08:51AM

This incident has been resolved.

MONITORING 12 months ago - at 05/22/2023 09:42PM

This issue has been resolved and we are currently monitoring.

MONITORING 12 months ago - at 05/22/2023 09:42PM

During a weekly update cycle, we implemented a modification to the Tempo system that inadvertently led to a significant increase in the number of series being produced by the Metrics Generator. This issue affected multiple regions for a duration of 6-11 hours. We have since identified the root cause and have deployed a fix to address it.

Latest Grafana outages

loki-prod & OLTP endpoint clusters are unreachable in us-east-0 region - 7 days ago
loki-prod & OLTP endpoint clusters are unreachable in us-east-0 region - 8 days ago
Issues Creating/Accessing Incidents. - 11 days ago
Grafana Cloud: prod-us-central-3 and prod-us-central-4 Clusters Unreachable - 17 days ago
AWS Germany -- Metrics Generator Issues - 23 days ago

Never miss when a third-party service is down

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3170 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime

Start today for FREE