Increased target registration - N. Virginia

We continue to make progress in processing the backlog of registration and deregistration events that were stuck between 7:58 AM to 9:10 AM PDT in the US-EAST-1 Region. We have now completed the vast majority of affected load balancers and expect new registration and deregistration operations to be operating normally. We have a small number of load balancers remaining that we continue to work on. ECS have also seen recovery for the vast majority of affected tasks. We anticipate that the remaining load balancers will recovery in or around the next 30 minutes and we will continue to provide updates as we progress in the remaining load balancers.

From 7:58 AM to 9:10 AM PDT we observed issues registering and deregistering load balancer targets to Application Load Balancers (ALBs) in the US-EAST-1 Region. Engineers were engaged at 8:10 AM and identified a backlog within the subsystem that handles ALB registrations and deregistrations. While we observed recovery for newly registered or deregistered targets starting at 9:10 AM, between 10:10 AM and 10:20 AM PDT we experienced additional impact that caused newly-registered load balancer targets to become stuck in the 'initial' state. We continue to process the backlog of registration and deregistration events that were stuck between 7:58 AM to 9:10 AM PDT and expect this to be completed within the next hour. ALB customers can unblock registration and deregistration attempts by retrying them. Some customers may be seeing impact with EKS or ECS container services for pods or tasks that attempted to scale to deploy between 7:58 AM to 9:10 AM PDT. As we work to process this backlog, we would expect these to resolve. We have messaged all affected customers via the Personal Health Dashboard during this event, but wanted to provide additional messaging on the Service Health Dashboard. We’ll continue to keep all messaging up-to-date as we proceed.