Blacksmith Outage History

GitHub experienced a degradation affecting GitHub Actions and Webhooks that lasted 2.1 hours. The Blacksmith service's GitHub webhook functionality was impacted during this period. The incident was resolved after GitHub addressed the underlying issues with their services.

Blacksmith's Actions service experienced degraded performance in the us-west region for 4.3 hours, with users reporting slow performance. A mitigation was applied and the service showed signs of recovery, with continued improvement observed during monitoring.

Blacksmith performed database maintenance that caused delays in job adoption for customers. The maintenance was completed within 2 minutes, but re-queuing existing jobs took approximately an hour while new jobs resumed normal processing. The incident lasted 3.9 hours total with minor impact on service availability.

GitHub Actions experienced a major degradation that lasted 7.4 hours, impacting the Blacksmith service. The incident was identified and monitored based on GitHub's official status reporting. The service team investigated the impact throughout the duration of the GitHub Actions outage.

GitHub Actions and Packages experienced degraded performance for 24 minutes due to a GitHub outage. The Blacksmith service was affected and monitored the situation as it developed. The incident was classified as minor and has since been resolved.

The Blacksmith service experienced sticky disk degradation in the US-WEST cluster, affecting Incremental Docker Builders for 4.2 hours. An emergency maintenance operation was performed on the storage cluster to resolve the issue. Docker build speeds improved following the maintenance and recovery efforts.

Blacksmith experienced a major control plane outage that affected job adoption for 3.5 hours. The incident was traced to a set of control plane ECS tasks that required mitigation. The service showed signs of recovery and was restored after monitoring confirmed the fix was effective.

The Blacksmith service experienced a control plane outage that affected job adoption functionality for 16 minutes. The incident was resolved and the service recovered to normal operation. The team monitored the recovery to ensure stability was restored.

A backend issue caused job queue backlog in the Blacksmith service, affecting job processing times for users. The team actively drained the accumulated queue over a 16-minute period until full resolution.

Blacksmith experienced elevated queue times for US x86 runners for 28 minutes on April 13th. The service team investigated the issue and queue times returned to expected levels. The incident was resolved through monitoring and intervention.