Slowness issue CI/CD runners

We are seeing a delay in jobs being scheduled on shared runners. For more information see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20287

The amount of queued jobs is steadily dropping, we continue working to identify the root cause. For more details please see : https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20287

We have identified some capacity issues in our Google Cloud infrastructure as the cause of this problem. For more information see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20287

We have contacted Google to check on the capacity issues. This is being investigated on their end. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20287.

We will now update this incident as we receive information from our colleagues at Google. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20287 for details.

As of 16:25 UTC our CI/CD job queue has gone back to operational levels. We continue to work with Google for any information on the cause. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20287 for details.

Job activity continues to be normal. We will monitor this for half an hour more and consider it resolved if the issue does not reoccur. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20287 for details.