PACE systems to be taken offline

Dear PACE users,

To address a security vulnerability, all PACE clusters (Phoenix, ICE, and Firebird) will be taken offline today, Monday, November 2, at 2:00 PM. Read below for more information.

For individuals who like to receive updates on PACE systems from https://status.gatech.edu/, the addition of new PACE categories requires current subscribers to resubscribe. To continue receiving notifications, visit the URL and click the "Subscribe" button in the top-right corner of the page.

Summary: At 2:00 PM today, the Phoenix, ICE, and Firebird clusters will be taken offline so that a security patch can be applied. Work will continue overnight to apply patches. We will notify users as access to each cluster is restored.

Details: PACE must apply a patch to fix a security vulnerability and keep our systems secure. To do this, every system has to be taken offline and re-imaged. The PACE team will work late into the evening and start early tomorrow morning to bring systems back online as quickly as possible.

Impact: This outage will impact all PACE clusters (Phoenix, Firebird, ICE), during which users will not be able to log in to any PACE cluster. During this period, access to CEDAR storage will remain available. Access to other data storage locations (e.g., project directories) will be temporarily unavailable; restoring access through Globus is a priority and we will notify users once data access is available for the Phoenix and ICE clusters. Firebird storage will not be accessible during the outage.

Jobs that have not started and will not finish prior to today at 2pm will be prevented from starting. Any jobs that have already started running and will be impacted by this downtime will be cancelled when the outage begins and will be refunded on December 1.

We apologize for this unexpected downtime and appreciate your understanding while we work to keep our systems secure. Please contact us at pace-support@oit.gatech.edu with any questions.

You can read this email on our blog, and status updates will be shared on https://status.gatech.edu/ as the work progresses.

Best,
The PACE Team

Dear PACE users,

Emergency maintenance on PACE systems will start at 2pm.

For more information on services impacted, please read the original announcement on the PACE blog.

Users will be notified as soon as individual clusters are fully restored. Additionally, information will be shared on the PACE blog and on https://status.gatech.edu/.

Best,
The PACE Team

Access to data on Phoenix and ICE using Globus is now available.

The status of each system has been updated.

Instructional cluster and Firebird compute nodes are passing health checks, with a few Phoenix nodes still pending. Additional storage, networking and user tests will begin shortly. Nexus cluster was patched and is back to normal operations.

The Phoenix and ICE clusters are now available. Users can log in and resume submitting jobs at any time.

Work on the Firebird cluster is still ongoing.