Lustre project file system issues on Phoenix cluster

Dear Phoenix Users,
Summary: /storage/coda1 is partially available from Phoenix cluster, affecting users with project directories in that file system (Lustre project storage).
Details: One of the metadata servers for the Phoenix Lustre project storage system rebooted overnight and has not yet returned to service. We are working to restore access.
Impact: Login via ssh to Phoenix may hang temporarily for all users but should eventually succeed. Researchers who have not migrated from Lustre project storage to VAST may not be able to access Phoenix OnDemand, and access to Lustre project storage may not be available from any Phoenix system. VAST project storage is not affected, nor is Lustre scratch storage.
Thank you for your patience as we work to resolve this issue. Please contact pace-support@oit.gatech.edu for any questions or concerns.
Best,
-The PACE Team

The Phoenix OnDemand server has become unresponsive, so all users will now be unable to reach it. The PACE team is investigating.
Access to the Lustre project storage filesystem has been restored, and it is now accessible from login and compute nodes. PACE continues to monitor the filesystem.

OnDemand Phoenix has returned to service.