AWS us-east-1 single availability zone outage

The upstream AWS incident in us-east-1 has been resolved by AWS, and all Buildkite services are operating normally. No further customer impact is expected. We appreciate your patience during this incident.

Despite the ongoing AWS incident, our own services are now stable. We are continuing to monitor our services closely, and are ready for further action should the need arise. We are also watching AWS services closely as they recover.

We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur.

(Apologies if you receive duplicated notifications for this update.)

We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur.

We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips will unfortunately continue while these manual failovers occur.

We are actively moving resources out of us-east-1c. Similar brief latency and error blips will be visible to customers while these manual failovers occur.

We have provisioned additional capacity in unaffected availability zones so that they are able to support the additional load. Automatic failovers continue to occur where necessary.

Some latency and transient errors will be visible to customers.

We are continuing to actively monitor the impacts of this availability zone outage for Buildkite customers. Some transient errors are visible due to availability zone failover events.

A small subset of our customers are experiencing delayed notifications. We are actively provisioning additional capacity for these customers.

Availability zone automatic failovers are occurring in response to the outage, and this is causing some brief error blips for some customers.

We're aware that AWS is reporting availability zone failures in us-east-1. We are monitoring the situation but so far there is no customer impact.