Incident with Webhooks and Actions

This incident has been resolved.

We are seeing recovery in Actions start times and are observing for any further impact.

We are still seeing about 5% of Actions runs taking longer than 5 minutes to start. We are scaling and shifting resources to encourage recovery of the problem.

We are still seeing about 5% of Actions runs taking longer than 5 minutes to start. We are evaluating mitigations to increase capacity to decrease latency.

We are seeing about 5% of Actions runs not starting within 5 minutes. We are continuing investigation.

We have seen recovery of Actions run delays. Keeping the incident open to monitor for full recovery.

Webhooks is operating normally.

We are seeing delays in Actions runs due to the recovery with webhook deliveries. We expect this to resolve with the recovery of webhooks.

Actions is experiencing degraded performance. We are continuing to investigate.

We are seeing recovery as webhooks are being delivered again. We are burning down our queue of events. No events have been lost. New webhook deliveries will be delayed while this process recovers.

Webhooks is experiencing degraded performance. We are continuing to investigate.

We are reverting a configuration change that is suspected to contribute to the problem with webhook deliveries.

Our telemetry shows that most webhooks are failing to be delivered. We are queueing all undelivered webhooks and are working to remediate the problem.

Webhooks is experiencing degraded availability. We are continuing to investigate.

We are investigating reports of degraded performance for Webhooks