Increased wait times for machine jobs

This incident has been resolved.

Machine, MacOS and Windows jobs are now within normal operating parameters. We will continue to monitor.

Machine and Windows jobs are now within normal operating parameters. Wait times continue to be longer than the expected normal for MacOS jobs as we work through the backlog of jobs. We thank you for your patience while our engineers continue to work towards mitigation.

We are continuing to optimize our infrastructure and resource usage due to a slower than expected recovery times. While the wait times have improved they continue to be longer than the expected normal. We thank you for your patience while our engineers continue to work towards mitigation.

We are beginning to see early signs of recovery with Linux and Windows jobs processing. As the system works through the backlog, you should start to see reduced wait times. Our engineers continue to monitor our resources closely to ensure a steady improvement in our systems. macOS jobs may see a longer recovery period.

We thank you for your patience during this time as we work towards a complete mitigation.

We are continuing to investigate and mitigate the high resource pressure on our platform. The mitigating efforts may lead to some steps within the running jobs having an empty output or being skipped. We thank you for your patience while our engineers work to bring the platform to a healthy state

We are continuing to alleviate the resource pressure our platform is currently experiencing. We thank you for your patience while we work to mitigate the issue causing high wait times on multiple job types

Our engineers continue to investigate the issue and are working on changes to our infrastructure to relieve the pressure caused by high resource usage. We thank you for your patience while we work to reduce these high queue times.

We are continuing to investigate this issue and are noticing high pressure on other job types at this time.

Our engineers are continuing to investigate the issue causing high utilization of our resources and higher wait times for machine and other job types. We thank you for your patience as we work to mitigate this issue.

Our engineers are currently investigating an issue causing longer wait times for machine jobs running on the platform. We will provide further updates as more information becomes available.