Metrics Write Outage in Multiple Cells

We are seeing improvement on the metrics side, with write performance recovering.

We continue to investigate the remaining impact to Synthetic Monitoring and are working to determine the underlying cause.

Monitoring will continue as recovery progresses, and we’ll provide further updates as we learn more.

Our teams have been alerted that Synthetic Monitoring will also be affected by this outage.

Users may see gaps in their Synthetic Monitoring metrics as well as missed alerts as a result of this.

We continue to investigate and will provide further updates as they become available.

We’ve re-evaluated the situation and this issue is still ongoing. Although we initially observed signs of recovery, write errors continue to occur in the affected cells.

Mitigation work is still in progress, and we’re treating the incident as identified again while we work toward a sustained resolution. We’ll provide further updates as we confirm stabilization.

Mitigation has been applied and Mimir write performance is beginning to recover in the affected cells.

prod-us-central-0.cortex-prod-10 appears to have recovered as of 19:52 UTC, and prod-us-central-5.cortex-dedicated-06 is showing signs of recovery as of 20:00 UTC.

We are continuing to monitor both cells closely to ensure the mitigation is effective and that the systems remain stable.

We are investigating a partial write outage affecting multiple metrics cells, beginning around 19:30 UTC. Some customers may see intermittent write failures or delays, but most requests should succeed after retries and recent metrics may appear late as a result.

Querying previously ingested data remains unaffected. Engineering is continuing to investigate and will provide further updates as more information becomes available.