Job stuck in pending state not being picked up by runner

We observe an increasing number of pipeline jobs stuck in pending state waiting to be pick up by runner. We're investigating the incident. See issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19969

Our teams are still investigating the problem. See more details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19969

We are seeing increasing numbers of pipeline jobs stuck in pending state waiting to be picked up by runners and have increased the severity. See issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19969

Our teams are still investigating the problem. See more details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19969

We have identified the problem and have made the relevant change. We can see jobs being picked up. See more details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19969

We can see jobs being picked up and executed and the runners processing the backlog. We are not yet back at full capacity. See more details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19969

There is a large backlog of jobs that runners need to work through. This will lead to delayed executions for some time. We are working on getting back to full capacity. See more details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19969

Our runner fleet is processing the large backlog. We are expecting delayed execution for some time. See more details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19969

Our runner fleet is catching up on the backlog. We are switching to monitoring. See more details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19969