Incident affecting Personal Environments for O11 and ODC

This incident has been fully resolved, and Personal Environments in the N. Virginia region for O11 and ODC are now operating as expected. Thank you for your patience.

Based on official feedback from AWS as of May 8th, the core issue impacting ODC is mitigated and the service is now operating normally.

For O11 Personal Environments, while the primary issue is resolved, some customers might still face 504 errors or access issues. A small number of instances remain impaired and AWS continues to work to recover all affected remaining resources. We will continue to closely monitor the situation and we will provide further updates when available.

We have now confirmation from AWS that access is now being restored on the affected environments. We suggest that customers can start testing on their end. Some customers might still face 504 errors or access issues. We continue to closely monitor the situation and we will provide further updates when available.

We have identified the root cause of the issue as a hardware failure within the AWS US East (N. Virginia) datacenter. We are actively deploying mitigations and closely monitoring AWS's recovery progress, which is expected to take several hours. Users may continue to experience symptoms while these fixes are being fully implemented.

We can now confirm that impact should only be perceived on O11 Personal Environments. AWS was able to identify that the issue was caused by a thermal event. Recovery might take several hours. We will continue to investigate this issue and closely monitoring the developments on the AWS investigations.

We are currently investigating an issue affecting O11 Personal Environments and ODC in the N. Virginia region. Customers may experience a variety of symptoms, most notably service unavailability for O11 and login/publishing issues for ODC. The cause has been identified to an issue in the AWS datacenter. We will provide an update as soon as more details are available.