Jobs stuck in running state

Between 16:20 and 16:32 UTC, job triggering and workflow starts experienced disruptions across all resource classes due to memory pressure on our internal job distributor systems. We identified the issue and scaled our infrastructure to handle the load. Services returned to normal operation at 16:32 UTC.

What was impacted: Job triggering and workflow starts were disrupted for 12 minutes. Some workflows and jobs appeared stuck in a running state during this window.

Resolution: Our systems are now operating normally with additional capacity in place to prevent similar disruptions. If you had workflows or jobs that were stuck during this window, please manually rerun them.

The incident is now resolved and we will be conducting a thorough review to understand what triggered the memory pressure and identify any additional preventive measures

As of 16:32 UTC, job triggering and workflow starts have returned to normal operation across all resource classes. The impact was limited to a 12-minute window between 16:20 and 16:32 UTC.

What's impacted: All new jobs and workflows are now starting normally.

What to expect: If you have workflows or jobs that were stuck during the 16:20-16:32 UTC window, please manually rerun them.

We are continuing to investigate the root cause of this disruption and will provide an update within 30 minutes or once our investigation is complete.

At 16:20 UTC, we began experiencing delays in job triggering and starts across all resource classes. Some workflows and jobs may appear stuck in a running state.

What’s impacted: Job triggering is experiencing delays or is stuck. This affects all resource classes and executors.

What to expect: If you have workflows that appear stuck and haven’t started, we recommend manually rerunning them.

We are actively investigating the root cause and working to restore normal processing speeds. Next update: We will provide an update within 30 minutes or earlier with our progress