PACE Enterprise Research Hall-Cooling issue

Databank is reporting an issue with the high temp condenser pump in the research hall of the CODA data center. The Research Hall is being power off in order for databank facilities to get the pump changed. This is not affecting the enterprise hall

Summary: A pump in the Coda Datacenter cooling system overheated on Saturday evening. All PACE compute nodes across all clusters (Phoenix, Hive, Firebird, Buzzard, and ICE) have been shut down until cooling is restored, stopping all compute jobs.

Details: Databank is reporting an issue with the high temp condenser pump in the Research Hall of the CODA data center, hosting PACE compute nodes. The Research Hall is being powered off in order for Databank facilities to get the pump changed.

Impact: All PACE compute nodes are unavailable. Running jobs have been cancelled, and no new jobs can start. Login nodes and storage systems remain available. Compute nodes will remain off until the cooling system is repaired.

Databank is estimating the repair of the cooling system in the Research Hall in the Coda DC to take 10 hours. During this time PACE will remain powered down.

Databank has reported that cooling has been restored to functionality and affected equipment has been replaced.

Cooling pumps were repaired over night. All PACE resources are being tested before getting the clusters back online.