S3 outage and High error rate in Frankfurt

This incident has been resolved.

The team is continuing to run repairs and monitor the situation. Some customers may notice some degraded service as things work themselves out and all systems return to normal. Customers may also notice that Backups and so on did not run overnight. As mentioned, the RCA will follow along with more information.

We are continuing to monitor for any further issues.

The issue has been resolved, and services should be returning to normal. The team is continuing to monitor the system recovery. We will publish the RCA as soon as the team has had a chance to review the records, and determine a cause

We are continuing to work on a fix for this issue.

The team has identified the source of the issue and has begun troubleshooting and remediation

Early indications point to an internal network issue, impacting communication between Buckets. External traffic is not impacted, and we do not see other customer-facing resources affected either.

The S3 team is currently working on an issue impacting S3 in Frankfurt. Based on the last reports, we are seeing High error rates likely caused by communication issues. we will update regularly as the issue is investigated and resolved.