Outage in AWS

Increased Error Rates and Latencies

Resolved Minor
June 13, 2023 - Started over 2 years ago - Lasted about 4 hours

Incident Report

We are investigating increased error rates and latencies in the US-EAST-1 Region.
Components affected
Amazon API Gateway Amazon API Gateway (us-east-1) Amazon AppStream 2.0 Amazon AppStream 2.0 (us-east-1) Amazon Athena Amazon Athena (us-east-1) Amazon Augmented AI Amazon Augmented AI (us-east-1) Amazon Braket Amazon Braket (us-east-1) Amazon Chime Amazon CloudFront Amazon CloudWatch Amazon CloudWatch (us-east-1) Amazon CodeGuru Profiler Amazon CodeGuru Profiler (us-east-1) Amazon CodeGuru Reviewer Amazon CodeGuru Reviewer (us-east-1) Amazon Cognito Amazon Cognito (us-east-1) Amazon Comprehend Amazon Comprehend (us-east-1) Amazon Connect Amazon Connect (us-east-1) Amazon DevOps Guru Amazon DevOps Guru (us-east-1) Amazon DocumentDB Amazon DocumentDB (us-east-1) Amazon ECR Amazon ECR (us-east-1) Amazon ECS Amazon ECS (us-east-1) Amazon Elastic File System Amazon Elastic File System (us-east-1) Amazon EKS Amazon EKS (us-east-1) Amazon ELB Amazon ELB (us-east-1) Amazon EMR Amazon EMR (us-east-1) Amazon ElastiCache Amazon ElastiCache (us-east-1) Amazon EventBridge Amazon EventBridge (us-east-1) Amazon FreeRTOS Amazon FreeRTOS (us-east-1) Amazon FSx Amazon FSx (us-east-1) Amazon GameLift Amazon GameLift (us-east-1) Amazon GuardDuty Amazon GuardDuty (us-east-1) Amazon Inspector Amazon Inspector (us-east-1) Amazon Interactive Video Service Amazon Interactive Video Service (us-east-1) Amazon Kendra Amazon Kendra (us-east-1) Amazon Kinesis Firehose Amazon Kinesis Firehose (us-east-1) Amazon Kinesis Video Streams Amazon Kinesis Video Streams (us-east-1) Amazon Lightsail Amazon Lightsail (us-east-1) Amazon Location Service Amazon Location Service (us-east-1) Amazon Managed Grafana Amazon Managed Grafana (us-east-1) Amazon Managed Service for Prometheus Amazon Managed Service for Prometheus (us-east-1) Amazon Managed Streaming for Apache Kafka Amazon Managed Streaming for Apache Kafka (us-east-1) Amazon Managed Workflows for Apache Airflow Amazon Managed Workflows for Apache Airflow (us-east-1) Amazon MemoryDB for Redis Amazon MemoryDB for Redis (us-east-1) Amazon MQ Amazon MQ (us-east-1) Amazon OpenSearch Service Amazon OpenSearch Service (us-east-1) Amazon Pinpoint Amazon Pinpoint (us-east-1) Amazon Quantum Ledger Database Amazon Quantum Ledger Database (us-east-1) Amazon Redshift Amazon Redshift (us-east-1) Amazon RDS Amazon RDS (us-east-1) Amazon Route 53 Amazon SageMaker Amazon SageMaker (us-east-1) Amazon SES Amazon SES (us-east-1) Amazon SQS Amazon SQS (us-east-1) Amazon Transcribe Amazon Transcribe (us-east-1) Amazon Translate Amazon Translate (us-east-1) Amazon WorkMail Amazon WorkMail (us-east-1) Amazon WorkSpaces Amazon WorkSpaces (us-east-1) AWS Account Management AWS Amplify Admin AWS Amplify Admin (us-east-1) AWS Amplify AWS Amplify (us-east-1) AWS AppSync AWS AppSync (us-east-1) AWS Batch AWS Batch (us-east-1) AWS Certificate Manager AWS Certificate Manager (us-east-1) AWS Cloud9 AWS Cloud9 (us-east-1) AWS CloudFormation AWS CloudFormation (us-east-1) AWS CodeCommit AWS CodeCommit (us-east-1) AWS CodePipeline AWS CodePipeline (us-east-1) AWS CodeStar AWS CodeStar (us-east-1) AWS Config AWS Config (us-east-1) AWS Control Tower AWS Control Tower (us-east-1) AWS Data Exchange AWS Data Exchange (us-east-1) AWS DataSync AWS DataSync (us-east-1) AWS Directory Service AWS Directory Service (us-east-1) AWS Elemental AWS Elemental (us-east-1) AWS Fault Injection Simulator AWS Fault Injection Simulator (us-east-1) AWS Global Accelerator AWS Glue AWS Glue (us-east-1) AWS Ground Station AWS Ground Station (us-east-1) AWS IAM AWS IoT SiteWise AWS IoT SiteWise (us-east-1) AWS Lake Formation AWS Lake Formation (us-east-1) AWS Lambda AWS Lambda (us-east-1) AWS License Manager AWS License Manager (us-east-1) AWS Management Console AWS Marketplace AWS Migration Hub Strategy Recommendations AWS Migration Hub Strategy Recommendations (us-east-1) AWS Organizations AWS Outposts AWS Outposts (us-east-1) AWS QuickSight AWS QuickSight (us-east-1) AWS Resource Groups AWS Secrets Manager AWS Secrets Manager (us-east-1) AWS Service Catalog AWS Service Catalog (us-east-1) AWS Single Sign-On AWS Single Sign-On (us-east-1) AWS Support Center AWS VPCE PrivateLink AWS VPCE PrivateLink (us-east-1) EC2 Image Builder EC2 Image Builder (us-east-1) Multiple services Multiple services (us-east-1)

Need to monitor AWS outages?

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Start Free Trial
Latest Updates ( sorted recent to last )
UPDATE over 2 years ago - at 06/13/2023 09:49PM

We are working to accelerate the rate at which Lambda asynchronous invocations are processed, and now estimate that the queue will be fully processed over the next hour. We expect that all queued invocations will be executed.

UPDATE over 2 years ago - at 06/13/2023 09:29PM

Lambda synchronous invocation APIs have recovered. We are still working on processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services (such as SQS and EventBridge). Lambda is working to process these messages during the next few hours and during this time, we expect to see continued delays in the execution of asynchronous invocations.

UPDATE over 2 years ago - at 06/13/2023 09:00PM

Many AWS services are now fully recovered and marked Resolved on this event. We are continuing to work to fully recover all services.

UPDATE over 2 years ago - at 06/13/2023 08:48PM

Beginning at 11:49 AM PDT, customers began experiencing errors and latencies with multiple AWS services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use by other AWS services. We have associated other services that are impacted by this issue to this post on the Health Dashboard.

Additionally, customers may experience authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also experience intermittent issues when attempting to call or initiate a chat to AWS Support.

We are now observing sustained recovery of the Lambda invoke error rates, and recovery of other affected AWS services. We are continuing to monitor closely as we work towards full recovery across all services.

UPDATE over 2 years ago - at 06/13/2023 08:38PM

We are beginning to see an improvement in the Lambda function error rates. We are continuing to work towards full recovery.

UPDATE over 2 years ago - at 06/13/2023 08:14PM

We are continuing to work to resolve the error rates invoking Lambda functions. We're also observing elevated errors obtaining temporary credentials from the AWS Security Token Service, and are working in parallel to resolve these errors.

UPDATE over 2 years ago - at 06/13/2023 07:36PM

We are continuing to experience increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. We have identified the root cause as an issue with AWS Lambda, and are actively working toward resolution. For customers attempting to access the AWS Management Console, we recommend using a region-specific endpoint (such as: https://us-west-2.console.aws.amazon.com). We are actively working on full mitigation and will continue to provide regular updates.

UPDATE over 2 years ago - at 06/13/2023 07:26PM

We have identified the root cause of the elevated errors invoking AWS Lambda functions, and are actively working to resolve this issue.

UPDATE over 2 years ago - at 06/13/2023 07:19PM

AWS Lambda function invocation is experiencing elevated error rates. We are working to identify the root cause of this issue.

UPDATE over 2 years ago - at 06/13/2023 07:08PM

We are investigating increased error rates and latencies in the US-EAST-1 Region.

Latest AWS outages

Increased Error Rates and Latencies - N. Virginia - about 1 month ago
Increased Error Rates and Latencies - N. Virginia - about 2 months ago
Increased Error Rates and Latencies - 2 months ago
SQS Increased API Error Rates - Stockholm - 2 months ago
Increased error rates - N. Virginia - 3 months ago

Status Aggregator for All Your Third-Party Services

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4600 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook