Outage in Elastic Cloud

Several customer facing deployments deleted across multiple regions

Resolved Minor
January 11, 2023 - Started almost 2 years ago - Lasted 1 day
Official incident page

Need to monitor Elastic Cloud outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Elastic Cloud, and never miss an outage again.
Start Free Trial

Outage Details

We have detected an automation failure that caused some very limited number of customer deployments to be incorrectly deleted between 2UTC and 3UTC on the 01/11/2023. The impacted deployments are spread over multiple regions.
Components affected
Elastic Cloud Deployment metrics: AWS us-east-1 Elastic Cloud Snapshot Storage Infrastructure (S3): us-west-1 Elastic Cloud Deployment logging: Azure azure-centralus Elastic Cloud Deployment logging: GCP us-east1 Elastic Cloud Watcher email actions Elastic Cloud Deployment snapshots: GCP europe-west3 Elastic Cloud Enterprise Search connectivity: GCP europe-west2 Elastic Cloud Enterprise Search connectivity: GCP us-central1 Elastic Cloud App Search connectivity: GCP us-central1 Elastic Cloud Kibana connectivity: Azure azure-centralus Elastic Cloud Watcher email actions Elastic Cloud Deployment snapshots: AWS us-west-1 Elastic Cloud APM connectivity: Azure azure-westeurope Elastic Cloud Deployment logging: AWS us-west-2 Elastic Cloud Kibana connectivity: Azure azure-westeurope Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-west-2 Elastic Cloud Deployment hosts: AWS us-east-1 Elastic Cloud Deployment metrics: GCP us-east1 Elastic Cloud Kibana connectivity: GCP asia-east1 Elastic Cloud Enterprise Search connectivity: GCP us-west1 Elastic Cloud Deployment snapshots: Azure azure-westeurope Elastic Cloud APM connectivity: Azure azure-eastus2 Elastic Cloud App Search connectivity: GCP europe-west3 Elastic Cloud Kibana connectivity: GCP europe-west2 Elastic Cloud Deployment logging: AWS eu-west-2 Elastic Cloud Elasticsearch connectivity: GCP europe-west3 Elastic Cloud Deployment hosts: AWS eu-west-2 Elastic Cloud Deployment snapshots: Azure azure-northeurope Elastic Cloud Deployment metrics: AWS us-west-1 Elastic Cloud Watcher email actions Elastic Cloud Elasticsearch connectivity: Azure azure-uksouth Elastic Cloud Elasticsearch connectivity: AWS us-east-1 Elastic Cloud Deployment hosts: Azure azure-westus2 Elastic Cloud Kibana connectivity: GCP us-central1 Elastic Cloud Deployment snapshots: Azure azure-uksouth Elastic Cloud Deployment snapshots: GCP europe-west2 Elastic Cloud APM connectivity: GCP us-west1 Elastic Cloud Enterprise Search connectivity: Azure azure-eastus2 Elastic Cloud Watcher email actions Elastic Cloud Enterprise Search connectivity: Azure azure-uksouth Elastic Cloud Deployment snapshots: GCP asia-east1 Elastic Cloud Deployment snapshots: GCP us-east1 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west2 Elastic Cloud Kibana connectivity: GCP us-west1 Elastic Cloud App Search connectivity: Azure azure-centralus Elastic Cloud Deployment hosts: GCP us-central1 Elastic Cloud Kibana connectivity: GCP europe-west1 Elastic Cloud Kibana connectivity: Azure azure-northeurope Elastic Cloud APM connectivity: GCP europe-west3 Elastic Cloud APM connectivity: AWS eu-west-2 Elastic Cloud Deployment metrics: Azure azure-centralus Elastic Cloud Watcher email actions Elastic Cloud APM connectivity: Azure azure-northeurope Elastic Cloud Kibana connectivity: AWS us-west-1 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): AWS us-east-1 Elastic Cloud APM connectivity: GCP europe-west1 Elastic Cloud App Search connectivity: Azure azure-uksouth Elastic Cloud Deployment hosts: GCP asia-east1 Elastic Cloud APM connectivity: AWS us-west-2 Elastic Cloud Watcher email actions Elastic Cloud APM connectivity: AWS us-west-1 Elastic Cloud App Search connectivity: AWS us-west-1 Elastic Cloud Deployment hosts: AWS us-west-1 Elastic Cloud Kibana connectivity: Azure azure-westus2 Elastic Cloud Watcher email actions Elastic Cloud Deployment metrics: GCP us-west1 Elastic Cloud Deployment snapshots: AWS us-west-2 Elastic Cloud App Search connectivity: GCP europe-west2 Elastic Cloud App Search connectivity: AWS us-west-2 Elastic Cloud Deployment snapshots: GCP us-west1 Elastic Cloud Elasticsearch connectivity: Azure azure-westus2 Elastic Cloud APM connectivity: GCP asia-east1 Elastic Cloud Deployment logging: Azure azure-northeurope Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west1 Elastic Cloud AWS EC2 Health: us-west-2 Elastic Cloud Enterprise Search connectivity: Azure azure-northeurope Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-east1 Elastic Cloud AWS EC2 Health: us-east-1 Elastic Cloud Elasticsearch connectivity: Azure azure-westeurope Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): GCP us-central1 Elastic Cloud Snapshot Storage Infrastructure (S3): eu-west-2 Elastic Cloud Deployment hosts: Azure azure-northeurope Elastic Cloud Kibana connectivity: Azure azure-uksouth Elastic Cloud Deployment logging: GCP europe-west3 Elastic Cloud APM connectivity: Azure azure-centralus Elastic Cloud Deployment logging: GCP us-central1 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): GCP us-west1 Elastic Cloud App Search connectivity: AWS eu-west-2 Elastic Cloud Deployment snapshots: Azure azure-eastus2 Elastic Cloud Deployment hosts: GCP us-west1 Elastic Cloud AWS EC2 Health: us-west-1 Elastic Cloud Deployment metrics: Azure azure-westeurope Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-uksouth Elastic Cloud Deployment logging: GCP europe-west1 Elastic Cloud Enterprise Search connectivity: GCP asia-east1 Elastic Cloud Watcher email actions Elastic Cloud Deployment metrics: AWS eu-west-2 Elastic Cloud Deployment snapshots: Azure azure-westus2 Elastic Cloud Deployment snapshots: GCP europe-west1 Elastic Cloud Enterprise Search connectivity: AWS us-west-2 Elastic Cloud Deployment hosts: GCP europe-west1 Elastic Cloud Deployment snapshots: AWS us-east-1 Elastic Cloud Watcher email actions Elastic Cloud Deployment logging: GCP us-west1 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-northeurope Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westeurope Elastic Cloud AWS EC2 Health: eu-west-2 Elastic Cloud Watcher email actions Elastic Cloud APM connectivity: Azure azure-uksouth Elastic Cloud Enterprise Search connectivity: AWS us-east-1 Elastic Cloud Deployment metrics: GCP europe-west1 Elastic Cloud Watcher email actions Elastic Cloud Enterprise Search connectivity: GCP europe-west1 Elastic Cloud Deployment metrics: Azure azure-eastus2 Elastic Cloud Elasticsearch connectivity: AWS us-west-1 Elastic Cloud Enterprise Search connectivity: Azure azure-centralus Elastic Cloud APM connectivity: AWS us-east-1 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-1 Elastic Cloud Deployment hosts: Azure azure-centralus Elastic Cloud Deployment metrics: AWS us-west-2 Elastic Cloud Deployment metrics: GCP asia-east1 Elastic Cloud Deployment metrics: GCP us-central1 Elastic Cloud Watcher email actions Elastic Cloud Enterprise Search connectivity: AWS us-west-1 Elastic Cloud Deployment logging: GCP europe-west2 Elastic Cloud Kibana connectivity: Azure azure-eastus2 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-eastus2 Elastic Cloud Deployment hosts: Azure azure-uksouth Elastic Cloud Deployment metrics: GCP europe-west3 Elastic Cloud Deployment logging: AWS us-west-1 Elastic Cloud Deployment hosts: GCP europe-west2 Elastic Cloud Elasticsearch connectivity: AWS us-west-2 Elastic Cloud Kibana connectivity: GCP europe-west3 Elastic Cloud APM connectivity: Azure azure-westus2 Elastic Cloud Snapshot Storage Infrastructure (S3): us-east-1 Elastic Cloud Enterprise Search connectivity: GCP europe-west3 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-2 Elastic Cloud Enterprise Search connectivity: AWS eu-west-2 Elastic Cloud Deployment snapshots: Azure azure-centralus Elastic Cloud Kibana connectivity: AWS us-east-1 Elastic Cloud Watcher email actions Elastic Cloud Deployment hosts: GCP europe-west3 Elastic Cloud App Search connectivity: Azure azure-westus2 Elastic Cloud App Search connectivity: GCP us-east1 Elastic Cloud Deployment metrics: Azure azure-northeurope Elastic Cloud Elasticsearch connectivity: GCP us-central1 Elastic Cloud Deployment hosts: AWS us-west-2 Elastic Cloud Enterprise Search connectivity: GCP us-east1 Elastic Cloud Deployment hosts: GCP us-east1 Elastic Cloud App Search connectivity: Azure azure-northeurope Elastic Cloud Kibana connectivity: GCP us-east1 Elastic Cloud Elasticsearch connectivity: GCP us-west1 Elastic Cloud Deployment logging: Azure azure-westeurope Elastic Cloud Watcher email actions Elastic Cloud Deployment metrics: Azure azure-uksouth Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west3 Elastic Cloud Deployment hosts: Azure azure-eastus2 Elastic Cloud Enterprise Search connectivity: Azure azure-westus2 Elastic Cloud Elasticsearch connectivity: Azure azure-eastus2 Elastic Cloud Deployment logging: Azure azure-westus2 Elastic Cloud Elasticsearch connectivity: Azure azure-centralus Elastic Cloud App Search connectivity: GCP europe-west1 Elastic Cloud Enterprise Search connectivity: Azure azure-westeurope Elastic Cloud Elasticsearch connectivity: GCP europe-west2 Elastic Cloud Kibana connectivity: AWS us-west-2 Elastic Cloud Deployment metrics: Azure azure-westus2 Elastic Cloud APM connectivity: GCP us-east1 Elastic Cloud APM connectivity: GCP europe-west2 Elastic Cloud Deployment snapshots: AWS eu-west-2 Elastic Cloud Kibana connectivity: AWS eu-west-2 Elastic Cloud App Search connectivity: Azure azure-westeurope Elastic Cloud App Search connectivity: AWS us-east-1 Elastic Cloud Deployment logging: Azure azure-eastus2 Elastic Cloud Elasticsearch connectivity: GCP asia-east1 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-centralus Elastic Cloud Deployment logging: Azure azure-uksouth Elastic Cloud Elasticsearch connectivity: Azure azure-northeurope Elastic Cloud APM connectivity: GCP us-central1 Elastic Cloud App Search connectivity: GCP asia-east1 Elastic Cloud App Search connectivity: Azure azure-eastus2 Elastic Cloud Deployment hosts: Azure azure-westeurope Elastic Cloud Watcher email actions Elastic Cloud Deployment metrics: GCP europe-west2 Elastic Cloud App Search connectivity: GCP us-west1 Elastic Cloud Elasticsearch connectivity: GCP us-east1 Elastic Cloud Deployment logging: GCP asia-east1 Elastic Cloud Elasticsearch connectivity: AWS eu-west-2 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westus2 Elastic Cloud Deployment logging: AWS us-east-1 Elastic Cloud Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east1 Elastic Cloud Elasticsearch connectivity: GCP europe-west1 Elastic Cloud Deployment snapshots: GCP us-central1 Elastic Cloud Watcher email actions Elastic Cloud Watcher email actions
Latest Updates ( sorted recent to last )
RESOLVED almost 2 years ago - at 01/12/2023 02:11PM

All remediating actions have been completed. Our support engineers are working directly with customers who were impacted during this incident.

Our team will follow up with a root cause analysis.

MONITORING almost 2 years ago - at 01/12/2023 06:16AM

Most deployments have been restored and we are monitoring closely. All impacted customers have been contacted and we're working closely with those impacted to ensure systems are functioning normally.

IDENTIFIED almost 2 years ago - at 01/12/2023 03:23AM

Restoration is still proceeding and support is reaching out directly to impacted customers with updates. We will continue to update with progress.

IDENTIFIED almost 2 years ago - at 01/12/2023 02:04AM

We have successfully restored a number of deployments. Restoration efforts are continuing, and we will continue to update as new information comes to light. Thank you for your patience.

IDENTIFIED almost 2 years ago - at 01/12/2023 12:34AM

We have begun restoring deployments from snapshot. We will update again in one hour.

IDENTIFIED almost 2 years ago - at 01/11/2023 10:50PM

Work continues on implementing the fix across regions. Another update will be sent in 1h.

IDENTIFIED almost 2 years ago - at 01/11/2023 09:44PM

Work continues on implementing the fix across regions. Another update will be sent in 1h.

IDENTIFIED almost 2 years ago - at 01/11/2023 07:47PM

Work continues on implementing the fix across regions. Another update will be sent in 1h.

IDENTIFIED almost 2 years ago - at 01/11/2023 05:58PM

Work continues on implementing the fix across regions. Another update will be sent in 1h.

IDENTIFIED almost 2 years ago - at 01/11/2023 04:23PM

Work continues on implementing the fix across regions. Another update will be sent in 1h.

IDENTIFIED almost 2 years ago - at 01/11/2023 02:08PM

Work continues on implementing the fix across regions. Another update will be sent in 1h.

IDENTIFIED almost 2 years ago - at 01/11/2023 12:48PM

We are progressing with the implementation of our fix and our engineers are focused on restoring any affected clusters.

IDENTIFIED almost 2 years ago - at 01/11/2023 12:03PM

We are continuing to work on a fix for this issue.

IDENTIFIED almost 2 years ago - at 01/11/2023 11:46AM

We have already reached out to all affected customers and are working on restoring the affected deployments.
Next update will be in 1hr.

INVESTIGATING almost 2 years ago - at 01/11/2023 08:55AM

We are continuing to investigate this issue.

INVESTIGATING almost 2 years ago - at 01/11/2023 08:28AM

We have confirmed which deployments got affected in which regions. We have blob storage snapshots and we will be attempting recovery from our backups.

INVESTIGATING almost 2 years ago - at 01/11/2023 08:26AM

We are continuing to investigate this issue.

INVESTIGATING almost 2 years ago - at 01/11/2023 08:15AM

We are continuing to investigate this issue.

INVESTIGATING almost 2 years ago - at 01/11/2023 08:15AM

We are continuing to investigate this issue.

INVESTIGATING almost 2 years ago - at 01/11/2023 08:05AM

We have detected an automation failure that caused some very limited number of customer deployments to be incorrectly deleted between 2UTC and 3UTC on the 01/11/2023. The impacted deployments are spread over multiple regions.

Start monitoring all your vendors in just 5 minutes

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3278 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime