Outage in Farm HPC cluster

Farm's slurmdbd having intermittent issues

Resolved Major

April 23, 2025 - Started 11 months ago - Lasted 5 months
Official incident page

Incident Report

Farm's slurmdbd is having intermittent issues. If you see an error like below, it means the problem has occurred again, and we will restart slurmdbd to bring it back into service. """sacctmgr: error: _open_persist_conn: failed to open persistent connection to host:monitoring-ib:6819: Connection timed out sacctmgr: error: Sending PersistInit msg: Connection timed out""" We have a support case open with SchedMD and will update this issue as we learn more.

Need to monitor Farm HPC cluster outages?

Monitor all your external dependencies in one place
Get instant alerts when outages are detected
Be the first to know if service is down
Show real-time status on private or public status page
Keep your team informed

Start monitoring for free

Latest Updates ( sorted recent to last )

RESOLVED 6 months ago - at 09/23/2025 12:42AM

This incident has been resolved.

INVESTIGATING 11 months ago - at 04/24/2025 12:22AM

Farm's slurmdbd is having intermittent issues. If you see an error like below, it means the problem has occurred again, and we will restart slurmdbd to bring it back into service.

"""sacctmgr: error: _open_persist_conn: failed to open persistent connection to host:monitoring-ib:6819: Connection timed out
sacctmgr: error: Sending PersistInit msg: Connection timed out"""

We have a support case open with SchedMD and will update this issue as we learn more.

Latest Farm HPC cluster outages

CSE subdomain offline. Farm available at farm.hpc.ucdavis.edu. - 13 days ago

Slurm not available - about 2 months ago

nas-6-1 still not functioning - 2 months ago

Quobyte Storage is Unavailable - 4 months ago

Farm not currently accepting logins - 6 months ago

Stop wasting hours on 'is it us or them?'

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 6020 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook