Outage in AWS

Increased Error Rates and Latencies

Resolved Minor
June 13, 2023 - Started over 1 year ago - Lasted about 4 hours

Need to monitor AWS outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including AWS, and never miss an outage again.
Start Free Trial

Outage Details

We are investigating increased error rates and latencies in the US-EAST-1 Region.
Components affected
Multiple services AWS Organizations Amazon SageMaker (us-east-1) Amazon Augmented AI Amazon Augmented AI (us-east-1) Amazon SageMaker Amazon Comprehend Amazon SQS (us-east-1) Amazon Comprehend (us-east-1) AWS Directory Service AWS Directory Service (us-east-1) AWS Outposts Amazon SQS AWS Outposts (us-east-1) Amazon Pinpoint (us-east-1) Amazon FreeRTOS (us-east-1) Amazon Kinesis Firehose Amazon Kinesis Video Streams Amazon EKS (us-east-1) Amazon ELB Amazon OpenSearch Service (us-east-1) Amazon WorkMail (us-east-1) Amazon Managed Service for Prometheus (us-east-1) AWS Certificate Manager Amazon Connect (us-east-1) Amazon Cognito Amazon EKS Amazon GameLift (us-east-1) Amazon ECR Amazon Location Service (us-east-1) AWS CodePipeline (us-east-1) AWS Cloud9 (us-east-1) AWS CloudFormation Amazon Pinpoint AWS Glue (us-east-1) AWS IAM AWS Elemental (us-east-1) AWS Lake Formation (us-east-1) Amazon SES (us-east-1) Amazon WorkSpaces AWS Certificate Manager (us-east-1) AWS Management Console Amazon Connect AWS Config (us-east-1) Amazon DocumentDB (us-east-1) Amazon Managed Streaming for Apache Kafka Amazon Managed Streaming for Apache Kafka (us-east-1) Amazon Managed Workflows for Apache Airflow Amazon Quantum Ledger Database Amazon RDS Amazon RDS (us-east-1) AWS Lambda AWS Service Catalog (us-east-1) AWS VPCE PrivateLink (us-east-1) Amazon WorkSpaces (us-east-1) Amazon ELB (us-east-1) Amazon Athena (us-east-1) Amazon DocumentDB Amazon ECR (us-east-1) AWS CodeStar (us-east-1) AWS VPCE PrivateLink AWS Cloud9 Amazon Inspector (us-east-1) Amazon Kinesis Video Streams (us-east-1) EC2 Image Builder (us-east-1) Amazon Lightsail (us-east-1) Amazon Location Service EC2 Image Builder Amazon Managed Grafana (us-east-1) Amazon MQ (us-east-1) AWS Batch AWS Lambda (us-east-1) Amazon FreeRTOS AWS Lake Formation Amazon WorkMail AWS Support Center Amazon AppStream 2.0 Amazon AppStream 2.0 (us-east-1) Amazon Braket Amazon Braket (us-east-1) Amazon Chime Amazon CloudFront Amazon CloudWatch Amazon CloudWatch (us-east-1) Amazon Cognito (us-east-1) Amazon EMR Amazon EMR (us-east-1) Amazon Athena Amazon EventBridge Amazon GameLift Amazon Inspector Amazon Kendra Amazon Kendra (us-east-1) Amazon SES AWS AppSync AWS AppSync (us-east-1) AWS CodePipeline AWS Service Catalog Amazon EventBridge (us-east-1) Amazon Lightsail Amazon Managed Service for Prometheus Amazon Managed Workflows for Apache Airflow (us-east-1) Amazon MQ Amazon OpenSearch Service AWS CloudFormation (us-east-1) AWS Config Amazon Managed Grafana AWS Fault Injection Simulator AWS Fault Injection Simulator (us-east-1) Amazon Kinesis Firehose (us-east-1) AWS Batch (us-east-1) Amazon API Gateway Amazon API Gateway (us-east-1) Amazon Quantum Ledger Database (us-east-1) Amazon Redshift Amazon Redshift (us-east-1) AWS CodeStar AWS DataSync AWS DataSync (us-east-1) AWS Elemental AWS Glue AWS IoT SiteWise AWS IoT SiteWise (us-east-1) Multiple services (us-east-1) AWS License Manager AWS License Manager (us-east-1) AWS CodeCommit AWS CodeCommit (us-east-1) AWS Migration Hub Strategy Recommendations AWS Migration Hub Strategy Recommendations (us-east-1) Amazon Interactive Video Service Amazon Interactive Video Service (us-east-1) Amazon FSx (us-east-1) Amazon Elastic File System (us-east-1) Amazon Elastic File System AWS Global Accelerator Amazon FSx Amazon GuardDuty (us-east-1) AWS Single Sign-On (us-east-1) AWS Data Exchange (us-east-1) AWS Single Sign-On Amazon GuardDuty AWS Data Exchange AWS Ground Station AWS Ground Station (us-east-1) Amazon Transcribe (us-east-1) Amazon Transcribe AWS Resource Groups AWS Account Management Amazon MemoryDB for Redis (us-east-1) AWS QuickSight AWS QuickSight (us-east-1) Amazon MemoryDB for Redis AWS Control Tower AWS Control Tower (us-east-1) Amazon ElastiCache Amazon ElastiCache (us-east-1) AWS Amplify Admin (us-east-1) AWS Amplify AWS Amplify (us-east-1) Amazon Route 53 AWS Amplify Admin Amazon DevOps Guru (us-east-1) Amazon DevOps Guru AWS Marketplace Amazon CodeGuru Reviewer Amazon CodeGuru Reviewer (us-east-1) Amazon CodeGuru Profiler Amazon CodeGuru Profiler (us-east-1) AWS Secrets Manager AWS Secrets Manager (us-east-1) Amazon ECS (us-east-1) Amazon ECS Amazon Translate Amazon Translate (us-east-1)
Latest Updates ( sorted recent to last )
UPDATE over 1 year ago - at 06/13/2023 09:49PM

We are working to accelerate the rate at which Lambda asynchronous invocations are processed, and now estimate that the queue will be fully processed over the next hour. We expect that all queued invocations will be executed.

UPDATE over 1 year ago - at 06/13/2023 09:29PM

Lambda synchronous invocation APIs have recovered. We are still working on processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services (such as SQS and EventBridge). Lambda is working to process these messages during the next few hours and during this time, we expect to see continued delays in the execution of asynchronous invocations.

UPDATE over 1 year ago - at 06/13/2023 09:00PM

Many AWS services are now fully recovered and marked Resolved on this event. We are continuing to work to fully recover all services.

UPDATE over 1 year ago - at 06/13/2023 08:48PM

Beginning at 11:49 AM PDT, customers began experiencing errors and latencies with multiple AWS services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use by other AWS services. We have associated other services that are impacted by this issue to this post on the Health Dashboard.

Additionally, customers may experience authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also experience intermittent issues when attempting to call or initiate a chat to AWS Support.

We are now observing sustained recovery of the Lambda invoke error rates, and recovery of other affected AWS services. We are continuing to monitor closely as we work towards full recovery across all services.

UPDATE over 1 year ago - at 06/13/2023 08:38PM

We are beginning to see an improvement in the Lambda function error rates. We are continuing to work towards full recovery.

UPDATE over 1 year ago - at 06/13/2023 08:14PM

We are continuing to work to resolve the error rates invoking Lambda functions. We're also observing elevated errors obtaining temporary credentials from the AWS Security Token Service, and are working in parallel to resolve these errors.

UPDATE over 1 year ago - at 06/13/2023 07:36PM

We are continuing to experience increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. We have identified the root cause as an issue with AWS Lambda, and are actively working toward resolution. For customers attempting to access the AWS Management Console, we recommend using a region-specific endpoint (such as: https://us-west-2.console.aws.amazon.com). We are actively working on full mitigation and will continue to provide regular updates.

UPDATE over 1 year ago - at 06/13/2023 07:26PM

We have identified the root cause of the elevated errors invoking AWS Lambda functions, and are actively working to resolve this issue.

UPDATE over 1 year ago - at 06/13/2023 07:19PM

AWS Lambda function invocation is experiencing elevated error rates. We are working to identify the root cause of this issue.

UPDATE over 1 year ago - at 06/13/2023 07:08PM

We are investigating increased error rates and latencies in the US-EAST-1 Region.

The best way to monitor AWS and all your third-party services

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3273 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime