HashiCorp Outage History

HashiCorp's apt repository at apt.releases.hashicorp.com experienced a major outage for 8.4 hours, preventing users from fetching and installing packages through the apt package manager. The issue affected all package installations from HashiCorp's official apt repository, disrupting software deployments and updates for users relying on this distribution method. HashiCorp's team identified the root cause and implemented a fix to restore full package installation functionality.

HashiCorp's HCP Terraform self-hosted agents failed to start after auto-upgrading from version 1.28.6+ to 1.28.9, causing run failures in HCP Terraform workspaces. The issue persisted for 16.6 hours and affected only self-hosted agents during this specific version upgrade path. HashiCorp resolved the problem by releasing version 1.28.10 and updating automatic upgrades to skip the problematic 1.28.9 version.

HashiCorp's Engineering team identified critical problems with Terraform 1.15.0 that affected certain configurations, causing issues for users running this version. The incident lasted 24.6 hours and was resolved with the release of Terraform 1.15.1, which included fixes for the known critical issues. Users were advised to either upgrade to 1.15.1 or downgrade to the stable 1.14.9 version as a workaround.

HCP Boundary experienced a control plane degradation that prevented users from creating new clusters or managing existing clusters through the portal for 7.6 hours. Existing Boundary clusters and active sessions remained fully operational throughout the incident. The engineering team identified the root cause and implemented a fix to restore full control plane functionality.

HashiCorp's GPG key rotation caused validation errors affecting Terraform (specifically users of Stacks functionality) and hc-install over a weekend period. The incident lasted 24.2 hours and impacted new installations that needed to install the Stacks Plugin. HashiCorp resolved the issue by releasing hc-install v0.9.4 and deploying fixes for Terraform with backports to versions 1.13, 1.14, and 1.15.

HashiCorp experienced intermittent connection resets that disrupted Terraform operations during init, plan, and apply phases for 4.5 hours. The engineering team implemented a fix to resolve the connectivity issues. The incident was classified as minor and the service was restored with ongoing monitoring.

HashiCorp experienced intermittent connection resets affecting Azure resource management operations and Azure provider downloads for 9 hours. The issue was caused by concurrency problems when communicating with Azure services. The incident was resolved through collaboration with provider teams to address the root cause and implement mitigation measures.

HCP Terraform experienced degraded performance for 38 minutes due to an unspecified issue. The engineering team implemented a fix to resolve the performance problems. The service was restored and monitored to ensure full resolution.

HCP Vault Dedicated customers experienced issues with disaster recovery cluster creation and management for 120.1 hours. Existing regional disaster recovery capabilities remained functional and unaffected during the incident. The team investigated the root cause while the service disruption was classified as minor impact.

HashiCorp's HCP Projects page experienced intermittent 404 errors and degraded performance for 1.5 hours. The engineering team identified the root cause and worked on implementing a fix. The issue has been resolved, restoring normal functionality to the Projects page.