GitLab pipelines experiencing longer than expected execution times

We are currently investigating reports of GitLab pipelines taking longer than usual to complete. Our team is aware of the issue and actively working to identify the cause of these delays. Issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

We are still investigating the cause of extended GitLab pipeline execution times. Our team continues to work on resolving this issue. We appreciate your patience and will update when we have more information. Issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

We hare still investigating the issue. The background processes may take longer to be picked up and executed than usual. This can result in emails not being delivered and commits not showing up in merge request activity. CI/CD jobs may be stuck in pending state. Issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

We are investigating issues with Sidekiq. See the issue for more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

No material updates at this time - our investigation into Sidekiq issues is ongoing. See the issue for more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

We are trying to isolate the problem to a number of background jobs contributing to this problem and we are taking them offline for some time. AI feature such as Duo Chat may also be impacted by this.
See the issue for more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

We are still working to isolate the problem to a specific background job. See the issue for more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

No current updates at this time. We are still working to isolate the problem but took measures to add relief. We will provide an update once the root cause has been identified. See the issue for more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

We are seeing improvements in background job processing and operations are recovering back to normal. Delays in email processing are still possible, and we are continuing with our investigation. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

No material updates at this time. We are continuing to investigate and monitor background job processing. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

Background job processing has been restored. We are continuing to monitor while we investigate further into the root cause. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

We are continuing to monitor. We recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

No material updates at this time. We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489