Partial Mimir Write Outage

This incident has been resolved.

Both read and write 5xx's and increased latency were experienced in the two periods:
23:56:15 to 00:32:45 UTC
00:55:30 to 01:36:15 UTC

Customers should no longer experience issues.

We will continue to monitor and provide updates.

We are continuing to investigate this issue.

Users may experience intermittent 5xx errors when writing metrics, though retries may eventually succeed, which can lead to delayed or missing data.

We continue to investigate and will update when we have more to share.

As of 00:28 UTC, we have observed improvement with the partial write outage. Customers should no longer experience issues with metrics ingestion.

We will continue to monitor and provide updates.

As of 23:57 UTC, our engineers became aware of an issue with prod-us-west-0 resulting in a partial write outage. Users may experience intermittent 5xx errors when writing metrics, though retries may eventually succeed, which can lead to delayed or missing data.

We continue to investigate and will update when we have more to share.