Outage in Google Cloud

Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy

Resolved Minor
March 17, 2023 - Started over 2 years ago - Lasted 3 days

Incident Report

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: Engineering team has mitigated the issue in asia-south1 region and currently working on rollback for all global regions starting at 2023-03-17 12:00 US/Pacific. The ETA for the rollback is not known at this time. We will provide more information by Friday, 2023-03-17 14:00 US/Pacific. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.
Components affected
Google Cloud Networking

Need to monitor Google Cloud outages?

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Try IsDown risk-free 14-day free trial · No credit card required

The Status Page Aggregator Built for IT Teams

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4522 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook