GitLab Outage History

GitLab experienced a 1.8-hour outage where CI jobs failed to process on GitLab SaaS Shared Runners, specifically affecting GitLab Hosted Runners and Hosted Runners on Linux. The issue was caused by a recent change that was identified and rolled back, restoring normal error rates and job processing. The backlog of queued jobs was expected to clear within 30 minutes after the fix was implemented.

GitLab experienced an issue where Duo Agent Platform Flows, including Duo Code Review, were not working for some projects. The root cause was identified and a fix was deployed over several hours. Error rates decreased significantly following the deployment, with the incident lasting 15.8 hours total.

GitLab experienced an issue where Duo Agent Platform Flows failed to start when using the default Docker image. The problem was identified and a fix was deployed to production over approximately 3.3 hours, with customers seeing improvements as the deployment completed.

GitLab Duo experienced access errors when users attempted to use `@GitLabDuo` mentions in merge requests, affecting GitLab.com, Dedicated, and Self-Managed instances. The team identified the root cause and deployed an initial fix via revert, but continued monitoring revealed ongoing failures for some users. A final fix was deployed after 10.8 hours that remediated the errors and brought error rates back to normal levels.

GitLab experienced git operation failures that prevented users from performing standard Git commands and repository interactions. The Gitaly team investigated the issue, identifying potential causes through log analysis, and implemented mitigation rules that successfully reduced the error rate. The incident was resolved after 2.4 hours of investigation and monitoring.

GitLab experienced delays in CI job processing where jobs remained pending while waiting for shared runners to become available. The incident affected the CI/CD GitLab SaaS Shared Runners component, preventing normal job execution for users. The team deployed a fix after 1.1 hours, restoring normal shared runner functionality and job processing.

GitLab Duo Agent Platform (DAP) experienced a major outage where customers were unable to access the service and received "Usage quota exceeded" errors. The issue was caused by a misconfigured feature flag on the Customers Portal, which was identified and disabled by the engineering team. The service was restored after 1.6 hours once caches refreshed, with customers confirming that DAP functionality had returned to normal.

GitLab experienced elevated error rates in their web service for 49 minutes, affecting multiple components including the website, API, Git operations, container registry, CI/CD runners, and various other services. The incident was classified as minor and the team investigated the root cause throughout the duration.

GitLab experienced CI/CD pipeline disruptions where jobs were stuck in pending status and not being executed due to runner pool exhaustion. The incident affected shared runners, private runners, and Windows shared runners across GitLab SaaS. The issue was resolved by adding more runners to the pool, with the incident lasting 36 minutes.

GitLab experienced degraded performance affecting Duo Agent Platform workflows across GitLab.com and GitLab Dedicated for 2.6 hours, causing code review and custom flows to fail with "Session failed" errors. The team identified the root cause, implemented a revert fix, and deployed the solution. Error rates returned to normal levels after the revert was deployed.