CI/CD pipelines intermittently stuck on running status

Some users are experiencing CI/CD pipelines where jobs and pipelines are stuck in a “running” state, even though earlier-stage jobs have completed successfully. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.

No material change since the last update. The incident remains under active investigation. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.

We’re still actively investigating and are currently collating data from affected pipelines to better understand the patterns and root cause. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.

We are currently reviewing logs from our production infrastructure and specifically the status and behavior of the pipeline workers for potential causes. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.

We’ve identified a potential issue with the job deferral mechanism that may be affecting pipeline transitions between stages. For details, please see https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462.

We have identified the cause of this incident as the recurrence of an existing defect in the pipeline job deferral mechanism. In this case, it was triggered by a period of temporary high pipeline concurrency. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.

At this time, we are confident that current pipelines are not being affected by this behavior. We are continuing to monitor new reports closely until the top of the hour to ensure there is no ongoing impact. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.