Outage in Grafana
Rare intermittent write failures in Tempo
Resolved Minor
Need to monitor Grafana outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Grafana, and never miss an outage again.
Start Free Trial
Outage Details
Users may receive internet write failures on collectors stating:
"Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp/cloud", "error": "rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway)"
The agent/collector will automatically retry the write attempt and succeed, so there is no loss in data collection.
Components affected
Grafana AWS US East - prod-us-east-0: Ingestion Grafana GCP US Central - prod-us-central-0: Ingestion Grafana GCP India - prod-ap-south-0: Ingestion Grafana AWS Singapore - prod-ap-southeast-1: Ingestion Grafana AWS Sweden - prod-eu-north-0: Ingestion Grafana AWS US West - prod-us-west-0: Ingestion Grafana GCP Belgium - prod-eu-west-0: Ingestion Grafana Azure Germany - prod-eu-west-1: Ingestion Grafana Azure US Central - us-central2: Ingestion Grafana Azure Netherlands - prod-eu-west-3: Ingestion Grafana GCP Brazil - prod-sa-east-0: Ingestion Grafana AWS India - prod-ap-south-1: Ingestion Grafana GCP UK - prod-gb-south-0: Ingestion Grafana AWS Australia - prod-au-southeast-1: Ingestion Grafana AWS Germany - prod-eu-west-2: Ingestion Grafana AWS Brazil - prod-sa-east-1: Ingestion Grafana GCP Australia - prod-au-southeast-0: Ingestion Grafana GCP Singapore - prod-ap-southeast-0: Ingestion Grafana AWS Canada - prod-ca-east-0: Ingestion Grafana US-CENTRAL: IngestionLatest Updates ( sorted recent to last )
RESOLVED
4 months ago - at 01/23/2025 11:04PM
This incident has been resolved.
MONITORING
4 months ago - at 01/22/2025 10:11PM
Multiple issues were identified with our rollout process that caused instability on the write path.
We have pushed out three fixes that drastically reduce the number of dropped writes and make all failed writes retryable. We will continue to closely monitor this issue and make improvements where necessary.
IDENTIFIED
4 months ago - at 01/22/2025 07:52PM
We are continuing to diagnose and improve this situation. Fixes have been rolled out on Monday January 20th, and today, Wednesday January 22nd, to reduce drops during rollouts.
IDENTIFIED
4 months ago - at 01/18/2025 12:16AM
The issue has been identified and a fix is being implemented.
INVESTIGATING
4 months ago - at 01/17/2025 07:56PM
We are continuing to investigate this issue.
INVESTIGATING
4 months ago - at 01/17/2025 07:56PM
Users may receive internet write failures on collectors stating:
"Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp/cloud", "error": "rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway)"
The agent/collector will automatically retry the write attempt and succeed, so there is no loss in data collection.
Latest Grafana outages
Instances Inaccessible in prod-us-central-0 - 5 days ago
Stop Juggling Dozens of Status Pages – Monitor Them All in One Place
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 4000 services available
Integrations with
