Pipelines using SaaS Runners stuck on Pending status

We are receiving reports of CI Pipelines failing to start and stuck on Pending status. This is only affecting pipelines using GitLab-hosted runners. A fix has already been identified and should be deployed to production soon. Please see https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18679.

We have performed a deployment rollback and previously stuck pipelines are being picked up. We will start a monitoring window for 30 minutes to confirm incident resolution. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18679.

Pipelines processed by GitLab-hosted runners are back to normal operation after the rollback. A revert MR is in process to be deployed to production which will address the root cause of the issue. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18679.

Next update for this incident will be once the revert MR is deployed to production. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18679 for more details.

We've discovered this incident also impacted self-managed Runners when attempting to verify their credentials on GitLab.com. 404 errors were generated, preventing job execution. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18679.