Databank Power/Cooling Issue

At around 0830 on Tuesday ATL-1 had power blip. It caused the GT High Temp Chiller to fail. The chiller has failed to restart. The DCI team is trying to determine why the chiller will not restart. This is affecting the research hall and PACE operations.

A cooling failure has impacted the Coda datacenter. Investigation as to the cause is underway.
To minimize impact and begin mitigating rising temperatures, PACE has initiated a partial shutdown. The Phoenix and Hive schedulers have been paused, and all idle compute nodes on Phoenix and Hive have been powered off. Running jobs are not currently impacted.
We will continue monitoring the situation and determine if additional measures are needed. ICE, Firebird, and Buzzard remain in production at this time.

Cooling in the datacenter has been restored. The PACE team is now working to resume access to start new jobs on nodes that were not powered off, then to power on the remaining nodes and verify their functionality before restoring Phoenix and Hive to full capacity.

The compute nodes that were not powered down have been released for new jobs, so queued jobs are now starting on these nodes (approximately 50% of Phoenix and 80% of Hive). PACE is now working to verify functionality of the nodes that were turned off this morning before returning them to service.