Network Connectivity Issues

We continue to see recovery of the issue causing increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We’re continuing to monitor network mapping latencies and will provide a final update once all has been confirmed to be operating normally.

We continue to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We have now seen network mapping propagation latencies return to normal levels in both usw2-az1 and usw2-az2 Availability Zones. At this stage, all EC2 instance launches and network configuration changes are operating normally. We’re continuing to monitor network mapping latencies and will provide a final update once all has been confirmed to be operating normally.

We continue to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We have now seen network mapping propagation latencies return to normal levels in usw2-az2. At this stage, all EC2 instance launches and network configuration changes in usw2-az2 are operating normally. We have seen strong recovery in usw2-az1 as well and will provide an update when network propagation latencies have returned to normal levels there as well.

We continue to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. Over the last 30 minutes, we have seen a strong improvement in network mapping latencies in both Availability Zones. We have also seen a reduction in latencies and error rates for other AWS services affected by this event. At this stage, we would expect that the vast majority of EC2 instance launches and network configuration changes would complete within the normal latencies. We continue to monitor until all network propagation latencies have returned to normal levels and will continue to provide updates.

We continue to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We continue to see improvement to network mapping latencies but they are not at normal levels yet. Other AWS services are also starting to see recovery as network mapping latencies improve. We will continue to keep you updated until network mapping latencies have returned to normal levels.

We continue to work toward resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We have now completed all updates to mitigate the resource contention being experienced by the subsystem responsible for network mapping propagation. We’re seeing early signs of improvements as the system works through the backlog of network mapping propagations. We expect to see network mapping propagation latencies continue to recover over the next 2 hours, but will keep you updated on progress.

We continue to work toward resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. While network mapping propagation times have remained stable, we have not yet seen the improvement in propagation latencies that we had hoped for. We continue to work on multiple paths to mitigation. We have completed the first phase to address the resource contention within this subsystem, and are actively working on the second and third phases in parallel. Our current plan is to implement these mitigations within a single AZ (usw2-az1), and verify recovery occurs before proceeding to perform these same mitigations in usw2-az2. We will continue to provide updates every 60 minutes, or as we have additional information to share.

We continue to work toward resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. While network mapping propagation times have remained stable, we have not yet seen the improvement in propagation latencies that we had hoped for. In parallel, we are working on several other updates to address the resource contention within the subsystem responsible for network mapping propagation. We will continue to keep you updated on our progress towards full recovery.

We continue to work toward resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We have successfully applied an update to the subsystem responsible for network mapping propagation to address resource contention. We have seen network mapping propagation times stabilize but they have not yet begun to trend towards normal levels. We expect that to begin over the next 30 minutes, at which time we expect latencies and error rates to improve. We will continue to keep you updated on our progress towards full recovery.

We continue to progress toward resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. At this time, we are approximately 50% completed with the update to address resource contention within the subsystem responsible for network mappings propagation in the usw2-az2 Availability Zone. Once we complete the update in usw2-az2, we will then move on to usw2-az1. Our current expectation is to have both Availability Zones fully resolved within the next 60 to 90 minutes, and we will continue to provide updates as recovery progresses.

We continue to make progress towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. In the last 30 minutes, we’ve continued applying an update to address resource contention within the subsystem responsible for network mappings propagation and are seeing early signs of improvement. We will continue to monitor before deploying this change more broadly and will continue to provide updates.

We continue to make progress towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. In the last hour, we applied an update to address resource contention within the subsystem responsible for network mappings propagation and are seeing early signs of improvement. We will continue to monitor before deploying this change more broadly and will continue to provide updates.

We continue to make progress towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. While we continue to make progress in addressing the issue, we wanted to provide some more details on the issue. Within Amazon Virtual Private Cloud (VPC) any changes to the network configuration - including launching an EC2 instance, attaching an Elastic IP address or Elastic Network Interface - needs to be propagated to the underlying hardware to ensure that network packets can flow between source and destination. We call this network configuration “network mappings”, as it contains information about network paths or mappings. Starting at 10:00 AM PDT this morning, we have been experiencing a delay in the propagation of these mappings within a single cell (part of the Availability Zone) in usw2-az1 and usw2-az2 Availability Zones. The root cause appears to be increased load to the subsystem responsible for the handling of these network mappings. We have been working to reduce the load on this service to improve propagation times, but while we have made some progress, mapping propagation latencies have not returned to normal levels. We continue to work to identify all forms of resource contention that could be leading to load, and have a few additional updates that we are currently working on.

Once we have reduced the load on the subsystem responsible for network mapping propagation, we would expect full recovery. Since the issue only affects networking mappings as a result of changes to the network configuration, avoiding any mutating changes to your network configuration in the affected Availability Zone would limit impact.

For customers looking to fail away from the affected Availability Zone or Region, please ensure that you are not relying on mutating network configuration in the affected Availability Zones as part of that process.

We will continue to keep you updated as we make progress towards full recovery.

We continue to make progress towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. The issue affects some instances in each of these Availability Zones, where network mappings are not being propagated to the underlying hardware. Any mutations (or changes) to the networking configurations - including launching an EC2 instance, attaching and Elastic IP address or Elastic Network Interface - would see delayed propagation times, which will affect network connectivity until completed. Other networking services like PrivateLink and API Gateway, are also seeing network connectivity issues where network mappings have been mutated. For other AWS services, such as Lambda and Container Services (ECS/EKS), delays in function and container creation, as well as increased error rates, may occur due to this issue.

We continue to see improvements in the usw2-az1 Availability Zone and continue to work on resolving the issue in the usw2-az2 Availability Zone. We recommend that customers avoid making mutating changes within the affected Availability Zones, which will ensure that no new network mappings need to be propagated to the underlying hardware, avoiding impact from this issue.

We will continue to keep you updated as we make progress towards full recovery.

We continue to investigate increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. The issue affects some instances in each of these Availability Zones, where network mappings are not being propagated to the underlying hardware. Any mutations to network configurations would see delayed mappings, which will affect network connectivity. Other networking services like PrivateLink and API Gateway, are also seeing network connectivity issues where network mappings have been mutated. For other AWS services, such as Lambda and Container Services (ECS/EKS), delays in function and container creation, as well as increased error rates, may occur due to this issue.

We have been working to resolve the issue and are seeing improvements in the usw2-az1 Availability Zone as a result of these efforts. We will continue to keep you updated as we make progress towards full recovery.

We can confirm increased networking latencies and errors affecting multiple Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We have identified a potential root cause of the errors and are attempting mitigations. Early signs are this mitigation is reducing error rates and latencies. We continue to work towards a full root case and recovery.

We are investigating increased networking latencies and errors affecting multiple Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region.