Cloud Services Outage

This incident has been resolved.

We are seeing recovery across most affected sites. While some services still remain partially degraded, we anticipate recovery to continue as platform health stabilizes.

Our platform team has taken several mitigative actions, but is still working on an overarching fix for this issue. Currently a configuration rollback is being initiated, and customers should begin seeing positive results around approximately 19:00 UTC, but full restoration will require further steps to normalize network traffic. Some configuration changes may be restricted while the platform team rolls out these fix efforts.
-----
Critical Note: The source of the issue has been confirmed to lie primarily within the Azure Front Door services, resulting in the primary impact of the outage being customer sites that significantly rely on Azure Front Door in their configuration.

Our platform team has taken several mitigative actions, but is still working on an overarching fix for this issue. Currently a configuration rollback is being initiated, and customers should begin seeing positive results around approximately 19:00 UTC, but full restoration will require further steps to normalize network traffic. Some configuration changes may be restricted while the platform team rolls out these fix efforts.

This has been identified by our cloud platform vendor as a DNS issue resulting in availability degradation of some services, including the service management portal. We have been informed that the vendor has taken remedial steps to address and restore portal access issues. The underlying issue is under investigation; further updates will be provided as progress is made.

We have identified an issue with our cloud hosting platform affecting a subset of customers, resulting in some sites failing to load. We are investigating and will provide updates as soon as they are available.