Job Execution Failure

Between approximately 10:00 and 13:30 UTC today, some customers experienced delayed or stuck job runs.

During a routine, provider-managed Kubernetes upgrade in one of our US regions, our cloud provider was unable to provision replacement compute capacity. Instead of aborting, the upgrade proceeded to take existing nodes out of rotation, leaving the region without capacity to start new jobs.

We rolled back the upgrade, restoring full scheduling capacity, and jobs are processing normally. We were forced to cancel older jobs that were unable to recover and recommend customers re-run any effected operations.

We are adjusting our upgrade configuration and maintenance windows to prevent this scenario from recurring.

We have identified the issue and pushed a change to resolve job processing. More details to follow once fully resolved.

We are investigating an issue affecting job runs for customers in our US region. A portion of scheduled and API-triggered job runs are failing to start. An affected run may remain in a Waiting state and then fail with the message "Run timed out. Restart the
run, and if this issue persists contact support." Runs that do start are completing normally.

Customers in our EU, Canada, and APAC regions are not affected. Our engineering team is actively investigating. We will post an update by 9:30 AM ET

We have identified an issue with jobs executing and investigating the root cause. We will update this incident when resolved.