Outage in Google Cloud

Vertex AI custom training jobs failing if using more than 2GB ephemeral storage

Resolved Minor
August 16, 2024 - Started 3 months ago - Lasted about 4 hours

Need to monitor Google Cloud outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Google Cloud, and never miss an outage again.
Start Free Trial

Outage Details

Summary: Vertex AI custom training jobs failing if using more than 2GB ephemeral storage Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Friday, 2024-08-16 17:00 US/Pacific. Diagnosis: Custom Vertex AI training jobs running on GKE and using more than 2GB of ephemeral storage may fail with the error ""Pod ephemeral local storage usage exceeds the total limit of containers 2Gi." Workaround: None at this time.
Components affected
Google Cloud Infrastructure Components Vertex AI Training Cloud Developer Tools Operations Cloud Machine Learning Vertex AI Training (us-west2) Vertex AI Training (europe-west6) Vertex AI Training (northamerica-northeast1) Vertex AI Training (us-central1) Vertex AI Training (asia-east1) Vertex AI Training (asia-northeast1) Vertex AI Training (us-east4) Vertex AI Training (europe-west2) Vertex AI Training (northamerica-northeast2) Vertex AI Training (europe-west3) Vertex AI Training (asia-south1) Vertex AI Training (europe-west4) Vertex AI Training (asia-northeast3) Vertex AI Training (europe-west1) Vertex AI Training (asia-east2) Vertex AI Training (asia-southeast1) Vertex AI Training (us-east1) Vertex AI Training (us-west1) Vertex AI Training (australia-southeast1)

Latest Google Cloud outages

Some ASM collection scan results may be delayed - 1 day ago
Multiple GCP Products including Google Cloud Networking, Virtual Private Cloud (VPC), Cloud Load Balancing are experiencing issues in the asia-southeast2 region - 5 days ago
We are investigating reports of issues with Google Cloud Support. - 6 days ago
An issue with Vertex Gemini API in multiple regions of Asia and America - 6 days ago
Issues opening cases - 6 days ago

The best way to monitor Google Cloud and all your third-party services

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3273 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime

Start today for FREE