[SMP] Service disruption in AU region

This incident has been resolved.

We are continuing to monitor for any further issues.

We are seeing signs of recovery for SMP in the AU region. We continue to monitor the situation.

Our SMP product is affected by the Azure incident in Australia East and is currently unavailable.

Impact Statement: Starting at approximately 08:30 UTC on 30 August 2023, a utility power surge in the Australia East region tripped a subset of the cooling units offline in one of the Availability Zones. While working to restore the cooling units, temperatures in the datacenter increased so we have proactively powered down a small subset of selected compute and storage scale units to avoid damage to hardware and reduce cooling system load. All impacted storage and compute scale units are in the same datacenter, within one of the region’s three Availability Zones (AZs). Multiple downstream services have been identified as impacted.

Current Status: We do not have an exact ETA at this time, but temperature in the impacted datacenter have been stabilized. The Azure service recovery process has commenced and is expected to progressively return over a number of hours. Due to the nature of this issue our storage scale units are expected to require additional recovery efforts to ensure all resources return in a consistent state. Note that any new allocations for resources will automatically avoid the impacted scale units.

We received the following update from our cloud provider Microsoft Azure. We continue to monitor the situation.

Impact Statement: Starting at approximately 08:30 UTC on 30 August 2023, a utility power surge in the Australia East region tripped a subset of the cooling units offline in one of the Availability Zones. While working to restore the cooling units, temperatures in the datacenter increased so we have proactively powered down a small subset of selected compute and storage scale units to avoid damage to hardware and reduce cooling system load. All impacted storage and compute scale units are in the same datacenter, within one of the region’s three Availability Zones (AZs). Multiple downstream services have been identified as impacted.

Current Status: We do not have an exact ETA at this time, but temperature in the impacted datacenter have been stabilized. The Azure service recovery process has commenced and is expected to progressively return over a number of hours. Due to the nature of this issue our storage scale units are expected to require additional recovery efforts to ensure all resources return in a consistent state. Note that any new allocations for resources will automatically avoid the impacted scale units.

We were notified by our cloud provider Microsoft Azure that they are currently experiencing a cooling issue in one of their datacenters in the Australia East region.

"We are experiencing impact related to a cooling issue a section of a datacentre in the Australia East region. The issue is resulting in connectivity and availability for services with dependencies on storage or compute in this section."

We are monitoring the situation.

We will send an additional update in 60 minutes.