Issue with CI jobs in pending state not being picked up

We are currently investigating issues with CI pipelines where jobs are in a pending state and not being picked up. More details about this incident can be found in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20442

We are seeing CI activity returning to normal and jobs are being processed again. We continue working to identify the root cause of the issue. More details about this incident can be found in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20442

CI activity is back to normal and the issue is no longer occurring. We continue working to identify the root cause of the issue. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/20442

We have identified saturation in one of our database replicas as the cause of this issue. We are adding additional database capacity to improve reliance while investigation continues. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20442 for details.

We're now in the process to increase database capacity in our infrastructure. We will update this incident when this action completes. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20442 for details.

The database infrastructure expansion has been completed and is now fully operational. We will now monitor this to make sure performance is as expected. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20442 for details.