Maintenance Mode Spikes (Global)

This incident has been resolved. Root cause was due to an errant application of additional security control policies that lead to a temporary failure of API calls across the platform. The change was applied at 14:26 UTC and reverted at 14:29 UTC. By 14:34 UTC, the reverted change was fully propagated. Services scaled to handle the increased throughput during recovery. All health checks were confirmed passing by 15:40 UTC.

All health checks are now passing and throughput has returned to normal. Errors with station association in Americas (US East) have cleared and throughput is decreasing. Dialer tests are now passing in Americas (US East). Monitoring all services to ensure continued stability. Next update 15 minutes.

Regions are passing health checks. Americas (US East) is still seeing errors with phone service (affecting new station association) and errors with dialer campaigns and call back services. Next update 15 minutes.

Services have largely recovered. Teams are monitoring health checks and determining if any additional actions need to be taken to speed resolution. Next update 15 minutes.

Increased maintenance mode errors have been identified globally. Root cause has been identified and mitigation steps applied. Affected services are in recovery. Next update top of the hour.