Outage in Grafana

EU West - Issues with Prometheus Read/Write Actions

Resolved Major
September 11, 2023 - Started about 2 years ago - Lasted about 3 hours
Official incident page

Incident Report

As of 14:33 UTC, we became aware of an issue affecting Hosted Metrics in the eu-west-2 cluster only. Users encountering this issue will experience 5xx errors when attempting to either write metrics or query metrics in their Hosted instances. Engineering is aware and actively investigating. We will continue to provide updates as information is shared.
Components affected
Grafana EU-WEST: Querying Grafana EU-WEST: Ingestion
The Status Page Aggregator Built for IT Teams

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Try IsDown risk-free 14-day free trial · No credit card required
Latest Updates ( sorted recent to last )
RESOLVED about 2 years ago - at 09/11/2023 05:51PM

As of 17:01 UTC, Engineering has applied a remediation and we continue to observe normal behavior for metric ingestion and querying. At this time, we are considering this incident resolved.

No further updates.

INVESTIGATING about 2 years ago - at 09/11/2023 03:11PM

As of 15:00 UTC, we have observed significant improvement in error rates and users should begin experiencing normal behavior when writing metrics and querying metrics. Engineering will continue efforts to ensure long term nominal behavior and we will continue to provide updates as we learn more.

INVESTIGATING about 2 years ago - at 09/11/2023 02:51PM

As of 14:33 UTC, we became aware of an issue affecting Hosted Metrics in the eu-west-2 cluster only. Users encountering this issue will experience 5xx errors when attempting to either write metrics or query metrics in their Hosted instances. Engineering is aware and actively investigating. We will continue to provide updates as information is shared.

Latest Grafana outages

New users registered in Synthetic Monitoring from October 15th are no able to create checks - about 21 hours ago
Synthetic Probe Connectivity Issues - 2 days ago
Loki degraded writes in SouthEast region - 3 days ago
Machine learning unavailable for the prod-us-west-0 region - 6 days ago
AWS Firehose Logs Down in Prod-Us-West-0 - 7 days ago

The Status Page Aggregator Built for IT Teams

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4522 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook