Astro UI and Worker Scaling Impacted by Authentication Provider Outage

Normal performance has returned to all Astro components.

We have seen some slowness in the UI as Astro recovers.

We have identified that the reason for task failures was that on Deployments with KubernetesExecutor and dag-only-deployments enabled, worker pods could not start up successfully and marked their tasks as failed. Similarly, CeleryExecutor Deployments on Runtime 8+ mark tasks as failed after being queued for 10 minutes, and so the slow execution speed from degraded auto-scaling caused some tasks to be marked failed.

The underlying provider outage appears to have ended. Access to Astro is now restored and appears to be functioning normally again. We did see that more tasks than usual failed during this period. We are continuing to investigate why these failures occurred. Please review your recent task runs as you may need to clear and restart tasks.

The authentication provider for Astro appears to be having a major outage and this is preventing access to Astro. This will not directly stop the successful execution of tasks, but it will impact auto scaling of workers, which could lead to much slower rates of task execution.