Pipelines not completing

We are currently investigating an issue where pipelines are not progressing between stages. This causes them to remain in a running state indefinitely, and merge requests are not being marked as ready. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676

We continue to investigate potential causes of this issue. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for the latest updates.

Work is still in progress to identify the cause of service degradation to pipelines. Update frequency will be increased to every 30 minutes or as soon as we have material updates to share. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for more information.

Work is ongoing to identify the root of this issue, including an analysis of our Redis infrastructure. Please continue to follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

Investigation continues. Remember restarting the pipeline or individual jobs may serve as a workaround. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

We are currently investigating errors on a specific Redis shard that may be contributing to the problem. For the latest updates and detailed information, please check https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

We're taking corrective actions to alleviate some unexpected pressure in one of our Redis shards, which may alleviate the problem. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for more details.

After alleviating the saturation reported in our Redis infrastructure we are seeing readings go back to healthy levels. We will monitor now for new occurrences and reports to confirm this is resolved. Please review https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

Despite resolving the Redis saturation issue, we're still receiving reports of hanging pipelines on GitLab.com. We understand the frustration this causes and are urgently investigating. Updates at https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

Reminder: If you have urgent pipelines affected by the ongoing issue, you can cancel and retry them. A retry may make the pipeline run properly. We're still working on a permanent fix. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

Investigation continues. Remember, restarting the pipeline on individual jobs may serve as a workaround. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

No material updates to report. We continue to investigate potential causes of this issue. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for the latest updates.

No material updates to report. We continue investigations and will provide further updates in 1 hour. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for the latest.

We have resolved the Redis saturation issue and see that all pipelines are functioning normally now. We will continue to monitor while continuing investigation into other aspects. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

We have resolved the Redis saturation issue and see that all pipelines are functioning normally now. We will continue to monitor while continuing investigation into other aspects. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

We have resolved the Redis saturation issue and see that all pipelines are functioning normally now. We will continue to monitor while continuing investigation into other aspects. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

We are moving this incident back to Investigating, as after resolving the Redis saturation, we are still seeing Pipelines not progressing between stages. Merge Requests are functioning normally now. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

Some Merge requests are still impacted by this incident. Merge requests are stuck with the message "Your merge request is almost ready!". The workaround is to recreate the impacted Merge Request. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

We have identified a potential commit that could have caused this incident. We are working on reverting it. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

We have enabled a feature flag that will prevent new pipelines and Merge Requests from being impacted. Pipelines and MR that were previously stuck will need to be recreated. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676