Outage in Astro

Worker Nodes Not Spinning Up in GCP Dataplane Clusters

Resolved Major
April 04, 2024 - Started 29 days ago - Lasted about 1 hour
Official incident page

Need to monitor Astro outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Astro, and never miss an outage again.
Start Free Trial

Outage Details

Incident Description: Some worker nodes within several GCP dataplane clusters are failing to spin up as expected. This issue is causing delays in task execution and may lead to DAGs/tasks getting stuck in the queued state or failing. Current Status: We have pinpointed the issue and confirmed its existence. Our engineering team is actively collaborating to resolve the problem. Impact: Delays in task execution within affected clusters. There is a risk of DAGs/tasks getting stuck in the queued state or failing due to the inability to spin up worker nodes. Resolution: Our engineering team is working diligently to implement a fix for this issue. Communication: Regular updates will be provided to keep you informed of any developments. We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve this issue promptly. Please stay tuned for further updates.
Latest Updates ( sorted recent to last )
RESOLVED 29 days ago - at 04/04/2024 04:52AM

This incident has been resolved.

MONITORING 29 days ago - at 04/04/2024 04:51AM

We are continuing to monitor for any further issues.

MONITORING 29 days ago - at 04/04/2024 03:56AM

A fix has been implemented and we are monitoring the results.

IDENTIFIED 29 days ago - at 04/04/2024 03:32AM

The issue has been identified and the fix is being implemented.

INVESTIGATING 29 days ago - at 04/04/2024 03:27AM

Incident Description: Some worker nodes within several GCP dataplane clusters are failing to spin up as expected. This issue is causing delays in task execution and may lead to DAGs/tasks getting stuck in the queued state or failing.

Current Status: We have pinpointed the issue and confirmed its existence. Our engineering team is actively collaborating to resolve the problem.

Impact: Delays in task execution within affected clusters. There is a risk of DAGs/tasks getting stuck in the queued state or failing due to the inability to spin up worker nodes.

Resolution: Our engineering team is working diligently to implement a fix for this issue.

Communication: Regular updates will be provided to keep you informed of any developments.

We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve this issue promptly. Please stay tuned for further updates.

Latest Astro outages

Authentication Errors - 16 days ago
Deployment metrics sometimes failing to load - about 1 month ago
Quay.io image registry is having an outage - about 1 month ago
Astronomer Cloud UI and API Unavailable - about 2 months ago
Astro CLI versions <= 1.22 are unable to successfully execute some commands - about 2 months ago

The easiest way to monitor Astro and all cloud vendors

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3154 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime

Start today for FREE