Incident Detected

Workspot has detected an incident. We are currently investigating and working to get the service up as soon as possible. Azure Service Disruption Summary of impact: The impact to customers is Control will timeout when trying to manage/add/change in East US2. Impact Statement: Starting at approximately 12:25 UTC on 08 Apr 2022, customers running services in the East US 2 region may be experiencing service management errors, delays, and/or timeouts. Microsoft is investigating an underlying issue causing GET and PUT errors impacting the Azure portal itself, as well as services including Azure Virtual Machines (VMs), Virtual Machine Scale Sets (VMSS), and additional downstream services. Customers may see errors including “The network connectivity issue encountered for Microsoft.Compute cannot fulfill the request.” Finally, for some downstream services that have auto-scale enabled, this service management issue may cause data plane impact. Current Status: Microsoft is investigating what is causing service management failures in this region. The Compute Resource Provider (CRP) which handles underlying compute resources for Virtual Machines is currently overloaded and unable to handle the volume of requests – causing the delays, timeouts, and errors described above. Microsoft has several parallel workstreams that aim to mitigate the issue – including scaling out the Service Fabric cluster from which the CRP gateway operates, rebooting Service Fabric nodes, and working with first-party services to reduce CRP traffic from the largest internal requestors. Microsoft is prioritizing workstreams that will mitigate customer failures, but also focused on understanding what is causing the CRP volume – including analyzing relevant memory dumps to understand the issues more deeply. The next update will be provided by 21:30 UTC, or as soon as there is an update to share.