Blacksmith Outage History

Blacksmith experienced degraded performance in GitHub Actions and Webhooks due to an upstream GitHub incident. The service disruption affected webhook ingestion and actions functionality for 28 minutes. The incident was resolved when the underlying GitHub issue was fixed.

Blacksmith experienced a control plane outage that affected job adoption capabilities for 1.9 hours. The incident impacted the API, website, and Blacksmith managed runners, preventing proper job processing. A fix was implemented and the service was restored to normal operation.

Blacksmith experienced elevated queue times for x86 runners for approximately 1.6 hours. The issue affected users' ability to quickly access x86 computing resources due to longer than normal wait times. The service was restored to normal queue times and the incident was resolved through monitoring.

ARM runners in the EU Central region experienced elevated queue times for 44 minutes. The issue affected job processing delays for users requiring ARM-based compute resources in that specific region. The incident was resolved after investigation and monitoring.

GitHub experienced degraded performance affecting Actions and other services, which impacted job adoption in the Blacksmith service. The incident was monitored and resolved after 56 minutes.

Blacksmith experienced elevated queue times in the EU-central region for 2.3 hours. The service team investigated the incident, which was classified as a minor disruption. The issue has been resolved.

Blacksmith experienced elevated queue times for x86 runners in the US region, causing delays for users trying to access these compute resources. The service disruption lasted 52 minutes while the team investigated the issue. The incident was classified as minor severity and has been resolved.

Blacksmith experienced delays in ARM job adoption specifically in the US region for 6.1 hours. The incident caused elevated queue times for ARM runners, affecting job processing performance. The issue was identified and investigated, with queue times remaining elevated during the incident period.

GitHub Webhooks experienced degraded performance for 1.1 hours due to an upstream GitHub incident. The Blacksmith service's webhook functionality was affected during this period. The issue was resolved when the underlying GitHub incident was fixed.

An upstream GitHub Actions incident caused instability in Blacksmith's service, affecting GitHub Actions, API requests, and webhooks with jobs taking longer than expected to be picked up. The service experienced elevated queue times in both regions during the 52-minute outage. Recovery was achieved once the upstream GitHub issue was resolved, though queue times remained elevated initially as the system recovered from the backlog.