Degraded performance for pipelines on gitlab.com

We have received reports of delayed or stuck pipelines, and jobs waiting longer than expected to execute. We are investigating. See issue for more details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

Investigation is still underway. An MR to increase resource limits to mitigate impact is currently awaiting merge. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

Resource limits have been increased and we are monitoring the impact. Investigation into the root cause is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

The resource limits increase did not sufficiently mitigate impact. Investigation into cause and mitigation options is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

Mitigation action has been taken and new pipelines can be triggered. Pipelines are being processed, but delays are still expected for up to 2 hours. Jobs may still be stuck in pending state during this time. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

We have identified the cause and taken action to mitigate re-occurance. We are seeing recovery on CI workloads and it should return to normal within the next hour. Some delays are still expected during this time.
See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

CI workloads are recovering as expected and should return to normal within the next hour. Some pipelines may still be delayed during this time. We are currently monitoring.
See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

We are continuing to see delays in CI job processing and are actively investigating mitigation steps. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

In addition to CI job processing delays, users may also see "Failed to retrieve CI/CD variables from cache" when running new pipelines manually via the UI. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

Issues with Hosted Runner CI job queue delays and cache errors related to CI/CD variables are still under investigation. Self-hosted Runners are not currently affected. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

No material update available at this time. Investigation into CI job delays on Hosted Runners and CI variable retrieval is ongoing. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

We have observed some improvement in queue times but are still investigating CI job delays on Hosted Runners and CI variable retrieval. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

We have observed that jobs on Hosted Runners are processing without delay and are able to retrieve cached CI/CD variables. We are currently awaiting deployment of a fix to ensure the issue remains mitigated. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

Hosted Runners continue to process without delay and are able to retrieve cached CI/CD variables. However some background processing is currently impacted and users may observe environments/review apps URLs not updating. We continue to wait for the fix to reach production. Details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

Background processing remains degraded, affecting Environments & Review Apps. Hosted Runners and CI jobs are healthy. A revert MR is queued for production, with more changes needed to address background processing. See: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

Background processing remains degraded, however the revert MR has been merged a short time ago. We are monitoring performance to see if the MR results in performance returning to optimal levels.
See: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

We've disabled the worker responsible for the issues. We've merged another fix for that specific worker and are in the process of deploying it to production before re-enabling it. The next update will be posted once that is completed.

Background processing remains degraded, affecting Environments and Review Apps, a fix is awaiting deployment. Hosted Runners and CI jobs are healthy. Details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

We have applied a fix and are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.