Massive underlying storage incident

The UAE1 region has been fully restored. All services are running normally. The on-call team is continuing a post-incident investigation.

Our engineering team has identified the root cause: a failure in the underlying storage infrastructure caused by memory pressure on storage components. We have applied a fix: storage component memory allocation has been significantly increased and a recovery deployment is actively rolling out. We are already seeing early signs of improvement, with storage components beginning to come back online.

Some processing services have been temporarily suspended to allow the storage layer to recover cleanly and safely.

We will post an update within 20 minutes or sooner if the situation changes.

We identified the problem, Ingestion is back, we are restoring the region.

We are continuing to investigate this issue.

We currently having issue on our underlying storage which impact several features. We are investigating.