Outage in Grafana

Issues with alert annotations in prod-eu-west-0 and prod-us-central-0

Resolved Minor
January 23, 2024 - Started 10 months ago - Lasted 1 day
Official incident page

Need to monitor Grafana outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Grafana, and never miss an outage again.
Start Free Trial

Outage Details

As of 1:00 UTC, we became aware of an issue with alerting in the prod-eu-west-0 and prod-us-central-0 clusters. Users experiencing this issue may encounter issues with alert annotations not showing up and other degraded alert features. Engineering is actively engaged and assessing the issue. We will provide updates accordingly.
Components affected
Grafana US-CENTRAL: Rule Evaluation Grafana EU-WEST: Rule evaluation Grafana EU-WEST: Alertmanager and Rules Configuration API Grafana US-CENTRAL: Alertmanager Grafana EU-WEST: Alertmanager Grafana US-CENTRAL: Alertmanager and Rules Configuration API
Latest Updates ( sorted recent to last )
RESOLVED 10 months ago - at 01/24/2024 02:40AM

Engineering has released a fix and as of 10:00 am UTC, customers should no longer experience any alerting issues.

IDENTIFIED 10 months ago - at 01/23/2024 02:53AM

We have partially worked out the issue, however, we had to scale down deployment in prod-us-central-0 and prod-eu-west-0. As a result of that, the previously mentioned issues with alert annotations not showing up and other degraded alert features have been mitigated, however, users may have trouble working with the SLO or Cloud Logs Exporter apps (creating/updating/deleting/listing objects).

INVESTIGATING 10 months ago - at 01/23/2024 01:29AM

As of 1:00 UTC, we became aware of an issue with alerting in the prod-eu-west-0 and prod-us-central-0 clusters. Users experiencing this issue may encounter issues with alert annotations not showing up and other degraded alert features.

Engineering is actively engaged and assessing the issue. We will provide updates accordingly.

Latest Grafana outages

Tempo Ingestion Disruption - about 4 hours ago
k6 browser tests aborted by system - 1 day ago
Grafana Cloud Prometheus - Unhealthy Ingesters - 7 days ago
Confluent Cloud Latency and High Error Rates - 10 days ago
New and recently unpaused/unarchived Grafana Cloud instances unable to start. - 15 days ago

The best way to monitor Grafana and all your third-party services

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3265 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime

Start today for FREE