Elevated Error Rates - APAC (Sydney)

This incident has been resolved.

All services remain healthy. We are continuing to monitor service health and processing of queued events. Calls and call controls are fully functional.

Service has recovered and backlog is being processed successfully. Test calls have completed and call controls are responsive.

While the service is more stable now, we are continuing to see instance timeouts on the cluster. Instance timeouts are preventing the service from effectively processing session queue events. This continues to impact customers ability to make and receive calls and use call controls. Development teams are implementing additional mitigation steps to address instance timeouts and improve processing.

As we continue to investigate, teams have scaled up instances on dependent services. With the help of additional CPU and memory, some circuit breakers have cleared, however, error rates remain elevated. Customers may still experience delays in calls connecting and slowness when using call controls such as mute and hold buttons.

Teams are continuing to work toward stability. There will still be delays in call controls until all error rates have cleared and the backlog is processed. Customers may also experience intermittent issues with call failures until all instances in the service are healthy again.

Customers may continue to see delays in call controls as the system continues to process backlogged events. Teams are continuing to implement additional mitigation steps to improve recovery.

Error rates across all services continue to decrease and services continue to process backlogged events. Customers may continue to see delays when making and receiving calls. Users may see delays with hold a mute buttons and agents may also be unable to hang up. Teams continue to implement mitigation steps to improve recovery.

Conditions have improved but we continue to see delays in event processing. Customers will continue to experience delays when making and receiving calls. Next update in 20 minutes.

We are continuing to see improvement. Additional mitigation steps are being implemented to speed recovery. As services continue to process backlog, customers will experience delays when making and receiving calls. Next update 15 minutes.

While we have seen some improvement, we are still seeing errors in region. Teams are continuing to investigate and are doubling capacity in region to aid recovery as services process backlogged events.

Mitigation steps have been implemented and we are beginning to see some recovery. Next update in 15 minutes

We continue to investigate root cause of the issue impacting inbound/outbound calling in Sydney region. Additional service teams have been brought in to aid in the investigation. Next update top of the hour.

Teams are continuing to investigate root cause of the issue impacting inbound/outbound calls in the Sydney region. Some mitigation steps are being implemented in an attempt to reduce error rates.

We are investigating elevated error rates in the Sydney region that are impacting inbound/outbound calls.