Increased error rate in US

Normal operation has resumed and been consistent for the last 30 minutes, so this incident is resolved.

About 1.75% of API calls between 10:08 and 10:38 received a HTTP 500 response due to database connections being refused.

A combination of high-load tasks, both from API calls and internal DB processes, all happened at once. This caused the database to degrade in performance and refuse some connections. Attempts to manually kill some of the processes did not succeed, for reasons we will investigate.

Once these jobs completed, performance recovered and normal operation resumed. Jobs which had failed and retried have since completed.

Database load is back to normal and error have ceased. We are continuing to monitor to ensure normal operation has resumed.

We are still seeing occasional errors via the API. These are in much less common and in smaller numbers, but not back to zero yet.
The underlying cause is above average load on our database, which we are working to resolve.

Error rates have returned to normal; we are continuing to monitor.

We're investigating an increase in API calls resulting in HTTP 500 errors