Phoenix Scheduler Outage

The Phoenix job scheduler is currently down, and PACE is investigating.

Unfortunately, the Slurm scheduler controller is down due to issues with Slurm's database and jobs are not able to be scheduled. We have submitted a high priority service request with SchedMD, the developer of Slurm, and should be able to provide an update soon.

Jobs currently running will likely run, but we recommend reviewing the output as there may be unexpected errors. Job waiting in queue will stay in queue until the scheduler is fixed.

The rest of the Phoenix cluster infrastructure (i.e. login, storage, etc.) outside of the scheduler should be working. We recommend not running commands that require interaction with Slurm (i.e. any scheduler commands like 'sbatch', 'srun', 'sacct', or 'pace-quota' commands, etc) because they will not work at this time.

We will provide updates soon as we work on fixing the scheduler.