CI/CD pipeline processing delays

We are seeing delays in CI/CD pipeline processing. The cause for this problem is under investigation. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21484 for more details.

Our team is still investigating this issue. Users might encounter some delays with processing CI/CD pipelines.

Our team is considering increasing resources to address the issue with the Sidekiq backlog impacting CI/CD pipelines. We're working on a deployment fix to address this, though we are seeing some improvement overall with queue durations.

Our team has turned off a specific worker to help improve pipeline processing. There has been significant recovery, and our team has identified this as a resource issue. We are monitoring for stability in addition to investigating the root cause and determining a permanent fix.

We are continuing to monitor but the main backlog drivers are mitigated. Users should expect normal operations. Our team is currently deploying a configuration update to further address this. We'll provide an update once it is deployed.