Network Delays Affecting MacOS Jobs

The macOS resources performance issue has been fully resolved.

All M1 and M4 Pro medium/large resources are operating normally with standard wait times and network performance. We have monitored the infrastructure for stability and confirmed full recovery of all affected services.

We apologize for any inconvenience this may have caused. Thank you for your patience during this incident.

We are observing recovery of the macOS resources performance issue.

Wait times for M1 medium/large and M4 Pro medium/large resources have returned to normal levels
Network performance within running jobs appears to be recovering

We are continuing to monitor the situation closely to ensure stability and will provide another update within 30 minutes as we confirm full resolution of the incident.

We are continuing to investigate degraded network performance affecting macOS resources in our infrastructure.

What's happening:
We have identified reduced network bandwidth that is causing performance degradation for specific macOS resource classes.

What's impacted:
M1 medium and large resources: Experiencing average wait times of up to 20 minutes before job start
M4 Pro medium and large resources: Experiencing average wait times of approximately 5 minutes before job start
Running jobs on affected resources: Network-dependent operations within jobs (cache restoration, pulling datasets, downloading dependencies) are also experiencing slowdowns.
All other macOS resource classes are operating normally

Next update:
We are actively working to resolve this issue and will provide an update within 30 minutes or sooner if the situation changes.

We are continuing to investigate this and are working with our datacenter provider to mitigate this issue.

We are experiencing delays starting jobs for M1 and M4 medium and large resource classes.
Running jobs on these resource classes will also experience delays due to degraded network performance.