Blacksmith Outage History

Blacksmith experienced job processing delays in the eu-west region due to a spike in jobs that caused temporary capacity shortage. The service team worked on re-balancing capacity to reduce the delays, while also monitoring a related GitHub Actions incident. The issue was resolved after 2.7 hours through capacity re-balancing efforts.

GitHub webhook delivery degradation caused jobs to queue up in the Blacksmith service for 8.8 hours. The upstream GitHub issue was resolved, but the system had to work through a substantial backlog of accumulated webhook events and queued tasks. Full recovery was achieved as the backlog was processed and runner pools returned to normal operation.

GitHub Actions experienced a degradation that lasted 8 minutes, affecting the Actions component of the GitHub service. The Blacksmith team identified the issue and monitored its impact while GitHub addressed the underlying problem. The incident was resolved after the brief 8-minute duration.

Blacksmith experienced job queueing issues due to a high volume of incoming jobs that exceeded system capacity. The incident caused delays in job adoption and processing. The service disruption lasted 1.1 hours and was classified as a minor incident.

The Blacksmith webhook service experienced a major outage lasting 3.1 hours. A fix was implemented within approximately 3 minutes of the initial investigation. The service was then monitored to ensure the resolution was effective.

A load-related issue in Blacksmith's control plane caused delays in job adoption for 3.8 hours. The engineering team implemented multiple fixes to address the performance bottleneck. The service was restored and monitored to ensure stable job processing.

Blacksmith experienced a major control plane outage lasting 6.6 hours due to issues with their upstream database provider. The incident caused delays with job adoption and created a backlog of jobs that needed to be re-queued. A fix was implemented and jobs were re-queued, but slow job adoption issues persisted while the team worked with the database provider to resolve the root cause.

Blacksmith experienced elevated queue times for Mac runners during a 4-minute service incident. The issue was classified as minor and was under investigation. The incident has been resolved.

Blacksmith experienced elevated latency in git checkout operations in the US West region due to an upstream network provider degradation. Some git checkout operations took significantly longer than normal baseline performance. The issue was resolved after 2.1 hours when the upstream provider fixed the network degradation and checkout times returned to normal.

Blacksmith experienced a load-related issue in the control plane that caused delays in GitHub workflow job adoption. The incident was classified as minor and lasted 1.4 hours. The team identified the root cause and worked on implementing a mitigation to resolve the delays.