Pipelines slow to start due to Google Container Registry latency issues

We continue to see slow image pulls from GCR. Google estimate a mitigation to be in place by Thursday, 2022-04-28 18:15 US/Pacific. Please follow their status page for updates at https://status.cloud.google.com/. Image pulls from Artifact Registry are not affected.

As a mitigation, we recommend either setting a max-time on steps, to prevent them consuming excess build minutes if you do experience a slow image pull, or transitioning to Artifact Registry, which is now the recommended way of storing container images - https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr.

Google have confirmed an issue with Google Container Registry that is causing high latency when pulling images. Please monitor their service status page at https://status.cloud.google.com/ for further updates.

We've identified that images hosted in Google Container Registry (gcr.io) are slow to pull, whilst images hosted on Artifact Registry (pkg.dev) are not impacted. We’ve contact Google for further information.

For anyone using gcr.io/google.com/cloudsdktool/cloud-sdk, you can swap to using the same image hosted at Dockerhub (https://hub.docker.com/r/google/cloud-sdk/).

We are investigating reports that some pipelines are slow to start, and stuck pulling images for several minutes.