Mimir Partial Write Outage

We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

Synthetic Monitoring has now also recovered. Customers should no longer experience alert rules failing to evaluate.

We continue to monitor for recurrence and will provide updates accordingly.

Engineering has released a fix and as of 22:25 UTC, customers should no longer experience ingestion issues. We will continue to monitor for recurrence and provide updates accordingly.

While investigating this issue, we also became aware that Synthetic Monitoring is affected. Some customers may have alert rules failing to evaluate.

As of 21:30 UTC, we are experiencing a partial ingestion outage in Grafana Mimir. This is affecting the write path, where some ingestion requests are failing or timing out.

Our engineering team is actively investigating and working to identify the root cause.