GitLab Outage History

GitLab.com experienced intermittent HTTP 50X errors that prevented users from accessing the website. The issue was caused by excessive traffic against specific endpoints, and GitLab implemented mitigation measures that successfully reduced error rates. The incident lasted 16 minutes and was fully resolved.

GitLab experienced a 2.6-hour incident where CI/CD pipelines intermittently became stuck in a "running" state despite earlier-stage jobs completing successfully. The issue was caused by a defect in the pipeline job deferral mechanism that was triggered during a period of high pipeline concurrency. The problem was resolved by addressing the deferral mechanism issue, with monitoring confirming that new pipelines were no longer affected.

GitLab's Duo Agentic Chat service experienced intermittent failures where users' messages would not receive responses for 3.3 hours. The issue was traced to authentication failures in the cloud connector service, initially suspected to be caused by a recent code deployment. The incident was resolved by deploying a patch that addressed the authentication problems, restoring stable service.

GitLab.com experienced intermittent HTTP 50X errors that prevented users from accessing the website. The incident affected the main GitLab website component and lasted for 36 minutes. The service was restored after the 36-minute outage period.

GitLab.com users experienced intermittent 400 Bad Request errors when attempting to log in, affecting some but not all users for 9 hours. The issue was traced to an upstream Cloudflare problem after GitLab's initial deployment rollback failed to resolve the errors. The incident was resolved once Cloudflare addressed their upstream issue.

GitLab experienced in-memory errors causing failures in remote flows, Duo Developer, and other foundational workflows for 2.1 hours. Users encountered "Session failed" errors when requesting code reviews through GitLab Duo and other affected services. The incident was resolved by reverting the changes that caused the root issue.

GitLab experienced increased 403 and 429 errors on Git HTTPS requests that affected some CI/CD pipelines for 1.6 hours. The issue impacted Git Operations, preventing some automated builds and deployments from completing successfully. GitLab resolved the incident by adjusting their protection rules and confirmed stability through monitoring.

GitLab.com experienced increased background processing delays that affected pipeline execution and merge trains for 25.7 hours. The delays impacted the website and background processing systems, causing users to experience slower pipeline execution and merge train functionality. Background processing largely recovered and stabilized, with the engineering team continuing to monitor the system for 24 hours to ensure no recurrence.