Outage in Google Cloud

Vertex AI custom training jobs failing if using more than 2GB ephemeral storage

Resolved Minor
August 16, 2024 - Started about 1 year ago - Lasted about 4 hours

Incident Report

Summary: Vertex AI custom training jobs failing if using more than 2GB ephemeral storage Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Friday, 2024-08-16 17:00 US/Pacific. Diagnosis: Custom Vertex AI training jobs running on GKE and using more than 2GB of ephemeral storage may fail with the error ""Pod ephemeral local storage usage exceeds the total limit of containers 2Gi." Workaround: None at this time.
Components affected
Google Cloud Infrastructure Components Vertex AI Training Cloud Developer Tools Operations Cloud Machine Learning Vertex AI Training (us-west2) Vertex AI Training (europe-west6) Vertex AI Training (northamerica-northeast1) Vertex AI Training (us-central1) Vertex AI Training (asia-east1) Vertex AI Training (asia-northeast1) Vertex AI Training (us-east4) Vertex AI Training (europe-west2) Vertex AI Training (northamerica-northeast2) Vertex AI Training (europe-west3) Vertex AI Training (asia-south1) Vertex AI Training (europe-west4) Vertex AI Training (asia-northeast3) Vertex AI Training (europe-west1) Vertex AI Training (asia-east2) Vertex AI Training (asia-southeast1) Vertex AI Training (us-east1) Vertex AI Training (us-west1) Vertex AI Training (australia-southeast1)

Need to monitor Google Cloud outages?

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Start Free Trial

Latest Google Cloud outages

We are investigating elevated error rates with multiple products in us-east1 - 4 months ago
Multiple GCP products are experiencing Service issues - 5 months ago
Google Compute Engine (GCE) issue impacting multiple dependent GCP services across zones - 6 months ago
Customers are experiencing connectivity issues with multiple Google Cloud services in zone us-east5-c - 7 months ago
Apigee customers may experience unable to login to Admin UI portal. - 8 months ago

The Status Page Aggregator Built for IT Teams

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4522 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook