Pantheon Outage History

Pantheon experienced a major incident where a high-risk configuration in the deployment workflow from Test to Live environments could have caused unintended data and file loss. The issue affected Workflow Operations and required an immediate freeze on all Live deployments to prevent potential data corruption. Engineers identified and implemented a fix within approximately 2 hours, restoring safe deployment capabilities after monitoring confirmed the resolution.

A small number of Pantheon customer accounts were compromised through credential stuffing attacks using passwords stolen from external data breaches unrelated to Pantheon. The unauthorized access potentially allowed attackers to modify customer site code and files. Pantheon worked directly with affected customers to secure their accounts and remediate unauthorized changes, with no further suspicious activity detected after the 30-hour incident.

Pantheon experienced a major Redis endpoint issue that disrupted connections for customer sites, primarily affecting the Europe region. The incident caused intermittent downtime for affected sites over a 28.2-hour period, with manual remediation available through support during the outage. A fix was implemented and monitored before the incident was fully resolved.

Pantheon experienced intermittent access issues affecting the Dashboard and Workflow Operations for approximately 107.5 hours. Users encountered performance problems and difficulty accessing the dashboard, with the engineering team recommending postponing non-critical dashboard and CLI tasks during the outage. The issue was resolved through multiple fix attempts and monitoring phases, with the platform returning to normal operations and confirmed stability.

Pantheon experienced intermittent access issues affecting the Dashboard, Workflow Operations, and Terminus Operations for 1.8 hours. Users encountered 502 responses, degraded performance, and failed commands when trying to access these services. The team implemented a fix and monitored the results before confirming full resolution of the incident.

Pantheon experienced degraded performance affecting Dashboard loading and Terminus-related workflows for 35.4 hours. The engineering team identified the root cause as issues with core data storage performance and asynchronous job runner behavior, requiring a deep investigation and fix implementation. The incident was resolved with all Dashboard and Terminus operations returning to normal functionality.

Pantheon experienced filesystem permission errors affecting select customer sites over a 154-hour period. The engineering team investigated the root cause while support staff provided manual remediation to restore functionality for affected sites upon request. The incident was resolved after the investigation was completed and appropriate fixes were implemented.

Pantheon experienced degraded performance and failures in platform workflows including code deploys, backups, multidev environment creation, and scheduled tasks for 13.7 hours. The incident affected users' ability to push code, create environments, and run development workflows, though site uptime remained unaffected. Engineers identified and implemented a fix, resolving the issue after monitoring confirmed the platform workflows were operating normally.