Increased API Errors

This incident has been resolved.

Our metrics show that machines API has mostly recovered. We're still working through a small number of degraded Managed Postgres clusters.

We have identified and implemented a fix for the root cause of this incident and are monitoring the results. Machines API should be seeing recovery.

On the Managed Postgres side, New MPG creates should be working. A small number of clusters remain degraded, or are experiencing connectivity issues. We are continuing to restore these as quickly as possible.

We are continuing to work on a fix for this issue. Users will still see elevated error rates creating new apps, machines, and MPG clusters, as well as other API operations. Users may also see newly created machines remaining in a `created` state for an extended period before starting.

We are continuing to work on a fix for this issue.

We are continuing to see elevated errors with creating new apps, new MPG clusters, and setting secrets. A small number of MPG clusters continue to see connectivity errors. We are continuing to work on these issues.

We continue to see increased error rates creating new apps and MPG clusters, as well as with setting secrets. We are continuing to work to resolve these issues.

A small number of MPG clusters are still experiencing connectivity issues. We are working to restore these to a healthy state as quickly as possible.

We have identified some continuing issues with creating new apps and secrets and are working to resolve them.

Our monitoring indicates that machines API, flyctl and dashboard access should have recovered. Some Managed Postgres clusters may still be affected -- Rest assured that your cluster's data is intact while we are working to recover them to a working state.

We have identified and implemented a fix for the API outage. Some Managed Postgres clusters may be affected as a secondary effect and we are currently working to restore them.

We are investigating an increase in errors that affects numerous API endpoints including the Machines API, flyctl and dashboard.