Outage in Grafana

Outage due to DNS problems on AWS

Resolved Major
February 18, 2025 - Started 10 months ago - Lasted about 5 hours
Official incident page

Incident Report

We are currently experiencing an outage on our instances locate on AWS cloud due to DNS problems. We are actively working to reestablish the service and quantify the whole impact of the issue. All our services running on this provider may be potentially affected.
Components affected
Grafana Cloud: Integrations Grafana US-WEST: Querying Grafana US-WEST: Ingestion Grafana US-EAST: Querying Grafana US-EAST Grafana US-EAST: Ingestion Grafana Cloud: AWS Logs Grafana AWS Australia - prod-au-southeast-1 Grafana AWS Australia - prod-au-southeast-1 Grafana AWS Australia - prod-au-southeast-1: Querying Grafana AWS Australia - prod-au-southeast-1: Querying Grafana AWS Australia - prod-au-southeast-1: Rule Evaluation Grafana AWS Australia - prod-au-southeast-1 Grafana AWS Australia - prod-au-southeast-1: Querying Grafana AWS Australia - prod-au-southeast-1 Grafana AWS Australia - prod-au-southeast-1: API Grafana AWS Australia - prod-au-southeast-1 Grafana AWS Australia - prod-au-southeast-1 Grafana AWS Australia - prod-au-southeast-1 Grafana AWS Australia - prod-au-southeast-1 Grafana AWS Australia - prod-au-southeast-1 Grafana AWS Brazil - prod-sa-east-1 Grafana AWS Brazil - prod-sa-east-1 Grafana AWS Australia - prod-au-southeast-1: Ingestion Grafana AWS Australia - prod-au-southeast-1: Ingestion Grafana AWS Australia - prod-au-southeast-1: Alertmanager and Rules Configuration API Grafana AWS Brazil - prod-sa-east-1 Grafana AWS Australia - prod-au-southeast-1: Ingestion Grafana AWS Brazil - prod-sa-east-1 Grafana AWS Australia - prod-au-southeast-1: Public Probes Grafana AWS Brazil - prod-sa-east-1 Grafana AWS Brazil - prod-sa-east-1 Grafana AWS Brazil - prod-sa-east-1 Grafana AWS Brazil - prod-sa-east-1 Grafana AWS Brazil - prod-sa-east-1 Grafana AWS Canada - prod-ca-east-0 Grafana AWS Canada - prod-ca-east-0 Grafana AWS Canada - prod-ca-east-0 Grafana AWS Brazil - prod-sa-east-1: Querying Grafana AWS Brazil - prod-sa-east-1: Querying Grafana AWS Australia - prod-au-southeast-1: Alertmanager Grafana AWS Canada - prod-ca-east-0 Grafana AWS Brazil - prod-sa-east-1: Querying Grafana AWS Canada - prod-ca-east-0 Grafana AWS Brazil - prod-sa-east-1: API Grafana AWS Canada - prod-ca-east-0 Grafana AWS Canada - prod-ca-east-0 Grafana AWS Canada - prod-ca-east-0 Grafana AWS Canada - prod-ca-east-0 Grafana AWS Germany - prod-eu-west-2 Grafana AWS Germany - prod-eu-west-2 Grafana AWS Germany - prod-eu-west-2 Grafana AWS Brazil - prod-sa-east-1: Ingestion Grafana AWS Brazil - prod-sa-east-1: Ingestion Grafana AWS Brazil - prod-sa-east-1: Rule Evaluation Grafana AWS Germany - prod-eu-west-2 Grafana AWS Brazil - prod-sa-east-1: Ingestion Grafana AWS Germany - prod-eu-west-2 Grafana AWS Brazil - prod-sa-east-1: Public Probes Grafana AWS Germany - prod-eu-west-2 Grafana AWS Germany - prod-eu-west-2 Grafana AWS Germany - prod-eu-west-2 Grafana AWS Germany - prod-eu-west-2 Grafana AWS Germany - prod-eu-west-4 Grafana AWS Canada - prod-ca-east-0: Querying Grafana AWS Canada - prod-ca-east-0: Querying Grafana AWS Brazil - prod-sa-east-1: Alertmanager and Rules Configuration API Grafana AWS Canada - prod-ca-east-0: Querying Grafana AWS Canada - prod-ca-east-0: API Grafana AWS Germany - prod-eu-west-4 Grafana AWS India - prod-ap-south-1 Grafana AWS India - prod-ap-south-1 Grafana AWS India - prod-ap-south-1 Grafana AWS India - prod-ap-south-1 Grafana AWS India - prod-ap-south-1 Grafana AWS India - prod-ap-south-1 Grafana AWS Germany - prod-eu-west-4 Grafana AWS Singapore - prod-ap-southeast-1 Grafana AWS Singapore - prod-ap-southeast-1 Grafana AWS Canada - prod-ca-east-0: Ingestion Grafana AWS Canada - prod-ca-east-0: Ingestion Grafana AWS Brazil - prod-sa-east-1: Alertmanager Grafana AWS Canada - prod-ca-east-0: Ingestion Grafana AWS Singapore - prod-ap-southeast-1 Grafana AWS Canada - prod-ca-east-0: Public Probes Grafana AWS Singapore - prod-ap-southeast-1 Grafana AWS Singapore - prod-ap-southeast-1 Grafana AWS Sweden - prod-eu-north-0 Grafana AWS India - prod-ap-south-1 Grafana AWS India - prod-ap-south-1 Grafana AWS India - prod-ap-south-1 Grafana AWS Singapore - prod-ap-southeast-1 Grafana AWS Sweden - prod-eu-north-0 Grafana AWS Germany - prod-eu-west-2: Querying Grafana AWS Germany - prod-eu-west-2: Querying Grafana AWS Canada - prod-ca-east-0: Rule Evaluation Grafana AWS Sweden - prod-eu-north-0 Grafana AWS Germany - prod-eu-west-2: Querying Grafana AWS Sweden - prod-eu-north-0 Grafana AWS Germany - prod-eu-west-2: API Grafana AWS Sweden - prod-eu-north-0 Grafana AWS Sweden - prod-eu-north-0 Grafana AWS US East - prod-us-east-0 Grafana AWS Singapore - prod-ap-southeast-1 Grafana AWS Singapore - prod-ap-southeast-1 Grafana AWS US East - prod-us-east-0 Grafana AWS US East - prod-us-east-0 Grafana AWS US East - prod-us-east-0 Grafana AWS Sweden - prod-eu-north-0 Grafana AWS Germany - prod-eu-west-2: Ingestion Grafana AWS Germany - prod-eu-west-2: Ingestion Grafana AWS Canada - prod-ca-east-0: Alertmanager and Rules Configuration API Grafana AWS Germany - prod-eu-west-2: Ingestion Grafana AWS US East - prod-us-east-0 Grafana AWS Germany - prod-eu-west-2: Public Probes Grafana AWS US East - prod-us-east-0 Grafana AWS US West - prod-us-west-0 Grafana AWS Sweden - prod-eu-north-0 Grafana AWS Sweden - prod-eu-north-0 Grafana AWS US East - prod-us-east-0 Grafana AWS US West - prod-us-west-0 Grafana AWS US West - prod-us-west-0 Grafana AWS Canada - prod-ca-east-0: Alertmanager Grafana AWS US West - prod-us-west-0 Grafana AWS US West - prod-us-west-0 Grafana AWS US West - prod-us-west-0 Grafana AWS US East - prod-us-east-0 Grafana AWS India - prod-ap-south-1: Querying Grafana AWS India - prod-ap-south-1: Querying Grafana AWS India - prod-ap-south-1: Querying Grafana AWS India - prod-ap-south-1: API Grafana AWS US East - prod-us-east-0 Grafana AWS US East - prod-us-east-2 Grafana AWS Germany - prod-eu-west-2: Rule Evaluation Grafana AWS US East - prod-us-east-2 Grafana AWS India - prod-ap-south-1: Ingestion Grafana AWS India - prod-ap-south-1: Ingestion Grafana AWS India - prod-ap-south-1: Public Probes Grafana AWS India - prod-ap-south-1: Ingestion Grafana AWS US East - prod-us-east-2 Grafana AWS US West - prod-us-west-0 Grafana AWS Singapore - prod-ap-southeast-1: Querying Grafana AWS Singapore - prod-ap-southeast-1: Querying Grafana AWS Germany - prod-eu-west-2: Alertmanager and Rules Configuration API Grafana AWS Singapore - prod-ap-southeast-1: Querying Grafana AWS Singapore - prod-ap-southeast-1: API Grafana AWS US West - prod-us-west-0 Grafana AWS US West - prod-us-west-0 Grafana AWS Singapore - prod-ap-southeast-1: Ingestion Grafana AWS Singapore - prod-ap-southeast-1: Ingestion Grafana AWS Germany - prod-eu-west-2: Alertmanager Grafana AWS Singapore - prod-ap-southeast-1: Ingestion Grafana AWS Singapore - prod-ap-southeast-1: Public Probes Grafana AWS Sweden - prod-eu-north-0: Querying Grafana AWS Sweden - prod-eu-north-0: Querying Grafana AWS Sweden - prod-eu-north-0: Querying Grafana AWS Sweden - prod-eu-north-0: API Grafana AWS India - prod-ap-south-1: Rule Evaluation Grafana AWS Sweden - prod-eu-north-0: Ingestion Grafana AWS Sweden - prod-eu-north-0: Ingestion Grafana AWS Sweden - prod-eu-north-0: Ingestion Grafana AWS Sweden - prod-eu-north-0: Public Probes Grafana AWS India - prod-ap-south-1: Alertmanager and Rules Configuration API Grafana AWS US East - prod-us-east-0: Querying Grafana AWS US East - prod-us-east-0: Querying Grafana AWS Singapore - prod-ap-southeast-1: Rule Evaluation Grafana AWS US East - prod-us-east-0: API Grafana AWS US East - prod-us-east-0: Ingestion Grafana AWS US East - prod-us-east-0: Ingestion Grafana AWS US East - prod-us-east-0: Ingestion Grafana AWS Singapore - prod-ap-southeast-1: Alertmanager and Rules Configuration API Grafana AWS US East - prod-us-east-0: Public Probes Grafana AWS US West - prod-us-west-0: Querying Grafana AWS US West - prod-us-west-0: Querying Grafana AWS Singapore - prod-ap-southeast-1: Alertmanager Grafana AWS US West - prod-us-west-0: Querying Grafana AWS US West - prod-us-west-0: API Grafana AWS US West - prod-us-west-0: Ingestion Grafana AWS US West - prod-us-west-0: Ingestion Grafana AWS Sweden - prod-eu-north-0: Alertmanager and Rules Configuration API Grafana AWS US West - prod-us-west-0: Ingestion Grafana AWS US West - prod-us-west-0: Public Probes Grafana AWS Sweden - prod-eu-north-0: Alertmanager Grafana AWS Sweden - prod-eu-north-0: Rule Evaluation Grafana AWS US East - prod-us-east-0: Rule Evaluation Grafana AWS US East - prod-us-east-0: Alertmanager and Rules Configuration API Grafana AWS US East - prod-us-east-0: Alertmanager Grafana AWS US West - prod-us-west-0: Rule Evaluation Grafana AWS US West - prod-us-west-0: Alertmanager and Rules Configuration API Grafana AWS US West - prod-us-west-0: Alertmanager

Need to monitor Grafana outages?

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Latest Updates ( sorted recent to last )
RESOLVED 10 months ago - at 02/18/2025 02:06PM

This incident has been resolved.

MONITORING 10 months ago - at 02/18/2025 01:20PM

Our engineering teams have applied fixes to tackle the issues, and all services on the AWS clusters should be operational.

We will continue monitoring them.

MONITORING 10 months ago - at 02/18/2025 01:06PM

Update in affected components: Prometheus Metrics on us-east-0 and services depending on this are back to operational.

MONITORING 10 months ago - at 02/18/2025 12:40PM

Update in affected components: Synthetics Monitoring components back to operational, services depending on Prometheus Metrics on us-east-0 still under examination.

We are continuing to monitor all the services across AWS clusters. Beware that some services may still have degraded performance until fully infrastructure stabilization.

IDENTIFIED 10 months ago - at 02/18/2025 12:23PM

We are continuing to work on a fix for this issue.

IDENTIFIED 10 months ago - at 02/18/2025 12:21PM

Update in affected components: IRM components partially recovered, Oncall services are fully operational, Incident services recovering.
Prometheus services are almost fully operational (monitoring recovery on us-east-0)

IDENTIFIED 10 months ago - at 02/18/2025 12:11PM

Update in affected components: Tempo services and asserts services have been restored, alerting services have been partially restored.

Currently monitoring all operative services.

IDENTIFIED 10 months ago - at 02/18/2025 11:53AM

We have identified the issue, and we are bringing back to operational state most of the services including: Loki services, Pyroscope services, and AI/ML Services. We are monitoring these services.

INVESTIGATING 10 months ago - at 02/18/2025 11:24AM

Update in affected components: OTLP Endpoint and Graphite proxy for querying and ingesting are fully operational.

INVESTIGATING 10 months ago - at 02/18/2025 10:57AM

We are continuing to investigate this issue and working on reestablishing the service.

INVESTIGATING 10 months ago - at 02/18/2025 10:14AM

Update in affected components: Hosted Grafana instances (stacks) are operational.

INVESTIGATING 10 months ago - at 02/18/2025 10:06AM

Update in affected components: Grafana Cloud k6 (and legacy app.k6.io) are fully operational.

INVESTIGATING 10 months ago - at 02/18/2025 10:01AM

We are continuing to investigate this issue and determining the full impact.

INVESTIGATING 10 months ago - at 02/18/2025 09:44AM

Update in components scope: potentially all our services running on AWS may be affected.

INVESTIGATING 10 months ago - at 02/18/2025 09:43AM

Update in components scope: potentially all our services running on AWS may be affected.

INVESTIGATING 10 months ago - at 02/18/2025 09:33AM

We are continuing to investigate this issue.

INVESTIGATING 10 months ago - at 02/18/2025 09:22AM

We are currently experiencing an outage on our instances locate on AWS cloud due to DNS problems.
We are actively working to reestablish the service and quantify the whole impact of the issue. All our services running on this provider may be potentially affected.

Status Aggregator for All Your Third-Party Services

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4600 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook