Jobs not starting

This incident has been resolved. Docker, Mac, Machine jobs should be running correctly.

Machine jobs are now running, we expect the backlog to clear over the next hour. We will continue to monitor.

We are seeing Docker and Mac jobs are now becoming operational. We are seeing Machine jobs are starting to run, but we are still monitoring for capacity issues.

We've identified a problem in our internal networking systems that triggered the issue and made changes to configuration and deployments to address the issue.

Docker jobs with contexts are starting to run successfully. We will continue to monitor for further issues with Contexts.
We are still working to add capacity for machine jobs.

UI access remains stable. We are still tuning capacity and resources to process backlogs. We appreciate your patience and understanding.

UI and API access has mostly recovered. We are still working through capacity issues for Machine jobs and Contexts.

We are continuing to add capacity to process the backlog of jobs. We appreciate your patience.

We are seeing jobs running again, and are adding additional capacity.

We are seeing intermittent successes in the UI as some components have been recovered. We are continuing to work on getting jobs moving and will update shortly with status on jobs.

Our debugging efforts have led to partial recovery of some internal services. We are seeing intermittent success on user actions and continue to work on restoration. We'll report back in under 20 mins as we see the impact.

We are continuing to see degradation on our services including jobs not starting and UI impacts. We are currently investigating networking issues and will update further within 30 mins.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue. Thank you for your patience while we work toward a resolution.

We are continuing to investigate this issue.

We are continuing our investigation. Currently jobs are delayed in starting and not completing.

We are seeing a delay in starting jobs. We are currently investigating and will update shortly.