Cloud Control Panel

From 20:32 - 20:52 UTC, users were unable to load Droplet and Volume paths in the Cloud Control Panel, as well as saw operations on Volumes failing, due to errors on the Volumes API.

We've confirmed full resolution of this incident and all services are functioning normally.

If you continue to experience issues or have any questions, please open a Support ticket from within your account.

Our Engineering team has confirmed the change has been reverted and all impact should now be mitigated. We are observing recovery and error rates dropping back to normal levels. At this time, users should be able to use both the Volumes API and the Cloud Control Panel normally.

We are monitoring the situation and will post an update once we confirm the issue is fully resolved.

Additional confirmed impact:

Users are unable to utilize the Volumes API, resulting in failures on CRUD (create, read, update, delete) operations for Volumes. Users will experience 500 errors via the API when attempting operations like create, attach, detach, delete, etc. Users with Kubernetes Clusters utilizing Volumes for storage will also see errors when attempting these operations.

Our Engineering team has confirmed the root cause of this issue to be an incorrect DNS change and is in the process of rolling that change back to mitigate impact. We've also confirmed users may experience the error when attempting to load the Networking tab in the Cloud Control Panel.

We will post an update as soon as the change has been reverted.

Our Engineering team is investigating an issue impacting loading of the Droplets and Volumes pages in the Cloud Control Panel. At this time, users will experience a "Looks like something went wrong..." error when attempting to visit the Droplets or Volumes tab, as well as when attempting to create Droplets or Volumes.

We will provide an update as soon as possible.