PHX1 Temperature Issue

Ambient temperatures continue to decrease steadily, and alarms are beginning to clear. It may take a couple more hours for temperatures to fully return to normal, however there is no customer impact. We are closing out this incident while we continue to monitor.

Temperatures are slowly dropping and we are continuing to monitor.

Our data center provider has provided this update:

PHX-01 is experiencing elevated white space temperatures caused by multiple utility bump events attributable to overnight storms in the area. We have on-site vendors working rapidly to resolve the situation and restore adequate cooling. Technicians have activated mitigation efforts to reduce rising space temperatures. Thus far we have restored Chiller D into operation as of 7:45 am local time. Additional remediation measures include the following action items (anticipated commencement local time; anticipated completion local time):

-Chiller C restoration (targeted 12:30 pm)
-Temporary on floor cooling and heat removal equipment added (approximately 10:00 am)
-Additional temporary chiller and temporary backup generator added (targeted approximately 4:00 pm)

Following the restoration of Chiller D, we have seen significant temperature reductions and expect that reduction to continue over the next 60 minutes.

Our data center provider is continuing to work with their vendor on a fix. We are continuing to see stable (but high) temperatures. There is currently no ETR.

Temperatures appear to have stabilized or improved slightly, however they are still elevated. The data center is continuing to work with their vendor on a fix.

All services that were experiencing a network outage are now recovering. Temperatures are still elevated, and we are continuing to monitor for updates, and are monitoring for any other impacts to our equipment.

Impacted services are being migrated to healthy network paths and should be recovering shortly. We are continuing to monitor for any additional alarms as our PHX data center works to resolve the cooling issue. The data center facilities team has HVAC vendors on site to assist with recovery.

Our Phoenix data center is experiencing increased temperatures, which is causing issues with a subset of network switches. This is impacting some of our services in the Phoenix data center. The data center operator is aware of the issue and is working on a fix. We will continue to monitor and provide updates as they are received.