Outage in AgResearch eRI
long GPFS waiters on compute-4 and compute-0
Resolved MajorIncident Report
We have found long waiters and a deadlock on compute-4. The node is being drained in Slurm now, in preparation for a GPFS restart there, maybe tomorrow. If further issues develop we made need to force that restart, which will kill all the jobs running on compute-4.
Trusted by 1,000+ teams
Never miss outages in third-party dependencies
Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.
Latest Updates ( sorted recent to last )
RESOLVED
9 months ago - at 07/29/2025 08:55PM
This incident has been resolved.
MONITORING
9 months ago - at 07/25/2025 12:23AM
GPFS has been restarted on compute-4. This has cleared the long waiters. compute-4 has now been resumed in Slurm
IDENTIFIED
9 months ago - at 07/25/2025 12:12AM
At AgR request we are killing the jobs on compute-4 and will restart GPFS there shortly. This is an attempt to clear the long waiters and reinstate compute-4 to Slurm
INVESTIGATING
9 months ago - at 07/24/2025 04:24AM
We have found long waiters and a deadlock on compute-4. The node is being drained in Slurm now, in preparation for a GPFS restart there, maybe tomorrow. If further issues develop we made need to force that restart, which will kill all the jobs running on compute-4.
Latest AgResearch eRI outages
Slow Access to Data on S: Datasets on ERI for several Users - 2 months ago
Windows clients hanging when accessing dataseta - 3 months ago
compute-[1-4] nodes affected by a GPFS long waiter - 4 months ago
No Internet connectivity from compute nodes - 4 months ago
Research Developer Cloud Dashboard and API Issue - 4 months ago
Never miss outages in third-party dependencies
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 6320 services available
Integrations with
