Cloud Control Panel and API

Our Engineering team has confirmed the full resolution of this issue, which impacted our Cloud Control Panel, API, Managed Kubernetes clusters, and Block Storage Volumes.

Between 16:47 - 17:18 UTC, users experienced 500 errors and/or latency when attempting to reach the Cloud Control Panel and API.

Between 16:47 - 18:49 UTC, users were unable to successfully attach/detach Block Storage Volumes.

Between 16:47 - 20:50 UTC, users were unable to create new Managed Kubernetes clusters, as well as a subset of users were unable to destroy or upgrade existing clusters.

If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel.

Our Engineering team has put further remediations in place and reconciliation efforts for Managed Kubernetes clusters have completed. At this time, creation for new clusters is enabled again and all cluster operations should be succeeding.

With the root cause identified and remediations in place, we are no longer observing any impact to users. We'll continue to monitor the situation and post an update once we confirm the issue is fully resolved. Thank you for your patience.

Our Engineering team continues work to remediate the root cause of this incident. Further changes have been made and Managed Kubernetes cluster creation will now fail outright should users attempt to provision a cluster. Attach and detach operations on Block Storage Volumes (except for Managed Kubernetes) are now available and are succeeding as expected.

Our Engineering team has identified additional impact related to the root cause of this incident. At this time, creation, deletion, and upgrading of Managed Kubernetes clusters may result in failure due to a failing dependency of control planes. Attach and detach operations on Block Storage Volumes also remain unavailable.

As a result of necessary remediation steps, attach and detach operations on Block Storage Volumes in all regions are currently unavailable. Our Engineering team is continuing work to fully remediate the root cause. The Cloud Control Panel and API remain accessible.

The implemented actions have been successful in remediating the impact of this incident and our Engineering team has confirmed error rates are returning to normal thresholds. At this time, the Cloud Control Panel and API are accessible again.

Our Engineering team has identified the cause of the issue with the Cloud Control Panel and API and is now taking steps to mitigate impact. We're seeing some recovery, but users may still be unable to reach the Control Panel or API. We'll post an update once we confirm the implemented actions have remediated impact.

We are investigating reports of issues reaching the DigitalOcean Cloud Control Panel and API. At this time, users may see 500 errors or latency when attempting to reach both the Control Panel and API. We'll provide an update shortly.