Elevated API Errors

This incident is resolved

We have fixed an error related to our Database replicas and everything is now operating normally. We're still monitoring the system for overall health.

We are seeing error rates spiking again. We are investigating

Our fix has been deployed and all systems are showing as currently healthy. New jobs should be able to be created, existing jobs should now be progressing normally, and webhook updates should all be firing. Queued courier and job updates are now being drained and sent.

We're performing another code rollback now which should then allow new jobs to progress normally. Existing job updates (courier updates and status updates) are being queued and will be sent once the system is healthy. Existing jobs have not been able to advance, but will do so once this rollback is complete.

We're experiencing an elevated level of API errors and are currently looking into the issue.