[RESOLVED] Elevated latencies for network change propagation
Jan 28 11:22 AM PST Between 7:08 AM and 10:48 AM PST, we experienced elevated latencies for network change propagation in the EU-WEST-1 Region. This was due to a delay in propagation of configuration updates on a sub-system which supports external and cross account connectivity. Customers experienced delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services were also impacted by this issue and were impacted by delayed or stuck workflows such as pulling container images.
Engineers were automatically engaged at 6:29 AM, prior to customer impact beginning. At 7:20 AM, root cause was identified. By 9:20 we began observing early signs of recovery. Our mitigation efforts completed at 10:43 AM and full recovery was observed at 10:48 AM. We recommend customers retry any failed requests. The issue has been resolved and all services are operating normally.
Update: 18:51 UTC - AWS is seeing significant signs of recovery and continue to work toward full resolution.
What’s Happening: We have received word that AWS is currently experiencing a networking issue in eu-west-1. AWS has confirmed elevated latencies for network change propagation in the EU-WEST-1 Region. This causes delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services are also impacted by this issue and may be impacted by delayed or stuck workflows such as pulling container images. Engineers with AWS were automatically engaged and are actively working on mitigating this issue.
AWS is already seeing early signs of progressive recovery. They will continue to work toward full recovery and will continue to provide updates as they work toward full mitigation. They recommend customers continue to retry failed requests. As recovery progresses, additional requests will succeed.
There are 22 affected AWS services:
AWS Deadline Cloud
AWS Elemental
AWS Fargate
AWS Global Accelerator
AWS NAT Gateway
AWS WAF
Amazon API Gateway
Amazon Athena
Amazon Cognito
Amazon DynamoDB
Amazon EMR Serverless
Amazon ElastiCache
Amazon Elastic Container Registry
Amazon Elastic Container Service
Amazon Elastic Kubernetes Service
Amazon Elastic Load Balancing
Amazon Elastic MapReduce
Amazon OpenSearch Service
Amazon Personalize
Amazon Relational Database Service
Amazon SageMaker
Amazon Simple Storage Service
Additional information and updates are available on the AWS Health Dashboard page: https://health.aws.amazon.com/health/status.
Impact to Clients: Potential issues in EU-WEST-1 affecting Amazon Elastic Compute Cloud (ECC)
Jan 28 10:45 AM PST We continue to work toward fully mitigating the issue that is resulting in elevated latencies for network change propagation delays in the EU-WEST-1 Region. We have a high degree of confidence that our current mitigation efforts will fully mitigate the issue. Our current estimation is that these mitigations will complete within the next 60 to 120 minutes. We recommend customers continue to retry failed requests. We will provide additional information as recovery continues to progress.
Environments Impacted: Networking Issue in EU-WEST-1 Region
Action Required: Informational Only
What’s Happening: We have received word that AWS is currently experiencing a networking issue in EU-WEST-1. AWS has confirmed elevated latencies for network change propagation in the EU-WEST-1 Region. This causes delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services are also impacted by this issue and may be impacted by delayed or stuck workflows such as pulling container images. Engineers with AWS were automatically engaged and are actively working on mitigating this issue.
AWS is already seeing early signs of progressive recovery. They will continue to work toward full recovery and will continue to provide updates as they work toward full mitigation. They recommend customers continue to retry failed requests. As recovery progresses, additional requests will succeed.
Additional information and updates are available on the AWS Health Dashboard page: https://health.aws.amazon.com/health/status
Impact to Clients: Potential issues in EU-WEST-1
Primary Contact: For HSP AWS Accounts, if you are experiencing impaired services, please contact AWS Support.
For all other Clients, contact HSP support by opening a ticket via the HealthSuite Support Portal or using one of the numbers below to contact our support line and speak to an engineer. HSP Support is available 24x7x365.
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 5850 services available
Integrations with