Connectivity Issues

11:25 AM PDT We are investigating an issue which is affecting network traffic for some customers using AWS services in the US-WEST-2 Region.

12:13 PM PDT We continue to investigate the issue affecting network connectivity within a single Availability Zone (usw2-az2) in the US-WEST-2 Region. While we continue to work towards root cause, we believe that the issue is affecting connectivity to Network Load Balancers from EC2 instances, connectivity from Lambda to EC2 instances and other AWS services, as well as connectivity between EC2 and some AWS services using PrivateLink. In an effort to further mitigate the impact, we are shifting some services and network flows away from the affected Availability Zone to mitigate the impact.

1:00 PM PDT We continue to investigate the issue affecting network connectivity within a single Availability Zone (usw2-az2) in the US-WEST-2 Region. We have narrowed down the issue to an increase in packet loss within the subsystem responsible for the processing of network packets for Network Load Balancer, NAT Gateway and PrivateLink services. The issue continues to only affect the single Availability Zone (usw2-az2) within the US-WEST-2 region, so shifting traffic away from Networking Load Balancer and NAT Gateway within the affected Availability Zone can mitigate the impact. Some other AWS services, including Lambda, ELB, Kinesis, SQS, RDS, CloudWatch and ECS, are seeing impact as a result of this issue.

2:26 PM PDT We have identified the root cause of the issue affecting network connectivity within a single Availability Zone (usw2-az2) in the US-WEST-2 Region and are actively working on mitigation. A component within the subsystem responsible for the processing of network packets for Network Load Balancer, NAT Gateway and PrivateLink services became impaired and was no longer processing health checks successfully. This resulted in other components no longer accepting new connection requests, as well as elevated packet loss for Network Load Balancer, NAT Gateway and PrivateLink endpoints. For immediate mitigation for NLB, customers should (1) disable ‘cross zone load balancing’ on Network Load Balancer, and then (2) deregister any targets that are in usw2-az2. For NAT Gateway/PrivateLink, you may modify your route tables to direct traffic to NAT Gateways in other Availability Zones or you may disable PrivateLink endpoints in usw2-az2.

3:23 PM PDT We are beginning to see signs of recovery, and continue to work toward full resolution.

4:02 PM PDT We have resolved the issue affecting network connectivity within a single Availability Zone (usw2-az2) in the US-WEST-2 Region. Beginning at 10:58 AM PDT, we experienced network connectivity issues for Network Load Balancer, NAT Gateway and PrivateLink endpoints within the US-WEST-2 Region. At 2:45 PM, some Network Load Balancers, NAT Gateways and PrivateLink endpoints began to see recovery and by 3:35 PM, all affected Network Load Balancers, NAT Gateways and PrivateLink endpoints had fully recovered. The issue has been resolved and the service is operating normally.