Delayed reads and writes in eu-central-1

This incident has been resolved.

We are continuing to monitor for any further issues.

The amount of time taken for writes to become readable has returned to the normal operating level. We will continue to monitor and update.

We are continuing to monitor for any further issues.

The region is currently experiencing elevated times for written information to become queryable along with elevated query run times. All written information is still being safely queued. We are working to minimize disruptions and we will continue to update as the situation evolves.

The storage engine has completed processing the bursted traffic. Writes are now queryable within the normal time ranges. Our team will continue to monitor.

The region received another burst of write traffic causing backup into the queueing system. Heavily bursting traffic has now been limited on a per organization basis. All organizations impacted by this limitation have been notified. The storage system is currently processing the bursted traffic. We will update here as the storage engine completes this processing.

We are seeing a return of the delayed reads and writes issue from a few hours ago. Our team is actively investigating.

We have identified the primary contributing factor for performance degradation (delayed writes and subsequent temporary read discrepancies). The impacted region has been receiving a large burst of writes twice daily which has saturated the storage layer. All writes to this region have been successfully recorded within our queuing system but are taking much longer than expected to become queryable. In response to this event, we have increased the resources available to the storage system. In addition, we have paused all tertiary background processes in the region to expedite recovery. We will continue to provide updates as the storage system recovers and will provide a complete root cause analysis when this incident has been resolved.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We have deployed some minor updates to the cluster but still continue to investigate the issue

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on this issue and a fix is being implemented.

The issue has been identified and a fix is still being implemented.

The issue has been identified and a fix is being implemented.

The AWS regions specified are experiencing delayed write/query operations and intermittent query failures.
We are continuing to investigate the issue.

We are continuing to investigate this issue.

We are currently investigating this issue.