Background processing degraded

Background task processing has remained normal and the incident is resolved.

Around 11:00 UTC, a change was deployed to production which inadvertently broke the background processing.

We received our first alert at 11:12 UTC. We reverted the offending change at 11:15 UTC, but this did not restore background processing completely.

Multiple changes were made to minimize the impact of the issue, which restored most functionality for newly created jobs by 11:54 UTC. This left a backlog of work queued for processing between 11:00-11:54 UTC stuck in an unprocessable state.

The stuck backlog work was due to requeue at around 13:00 UTC due to reaching a configured timeout of 2 hours. We attempted to find a way to process this work sooner but were unsuccessful. As anticipated, the work became available from around 13:00 UTC and was processed for about half an hour, completing by 13:30 UTC.

This fully restored normal operations.

A postmortem of the incident will take place and be attached to this incident in the next 48 hours. If you have any queries in the interim, please contact us at support@cronofy.com.

The backlog of jobs has been processed and we are monitoring to verify that everything remains normal.

New work continues to process as normal, and the stuck jobs are expected to be processed in the next hour.

Capacity has been increased but the stuck jobs are still not processing due to an issue with the queue. We are working to find a way to process these jobs. New jobs are avoiding the stuck queue and are processing as expected.

We have mitigated most of the impact for background processing but have jobs stuck in US & DE. We are working to further increase capacity and execute these jobs.

Processing has recovered in all data centers apart from US & DE, which have one partly recovered. We are working to mitigate the issue in these two data centers.

We have seen background processing degrade in all data centers after a recent deployment. We have reverted the change and are investigating the cause.