GitLab Outage History

GitLab experienced degraded performance affecting Duo Agent Platform workflows across GitLab.com and GitLab Dedicated for 2.6 hours, causing code review and custom flows to fail with "Session failed" errors. The team identified the root cause, implemented a revert fix, and deployed the solution. Error rates returned to normal levels after the revert was deployed.

GitLab experienced a high backlog of jobs that caused pipelines and CI/CD jobs to run slower than normal. The service disruption lasted approximately 1.2 hours and was classified as a minor incident. The issue has been resolved, restoring normal pipeline and CI/CD job processing speeds.

GitLab.com experienced elevated 5xx errors for 28 minutes, affecting the website and Git operations. The issue was identified as a frozen Gitaly node, which GitLab worked to correct. Users experienced general service issues during this period.

GitLab.com experienced elevated error rates for 3.5 hours due to a major Redis Sidekiq cluster outage, causing intermittent failures and degraded performance across the website, API, Git operations, CI/CD pipelines, container registry, and authentication services. Users experienced delayed or non-running pipelines and authentication failures. The issue was resolved through targeted restarts and forced resets of the Redis cluster, with error rates returning to normal levels.

GitLab experienced 500 errors when accessing repositories and Gitaly errors during push operations, affecting both the website and Git operations for 2.6 hours. The incident impacted users' ability to access and push code to repositories. The service was restored by restarting the Gitaly process.

GitLab experienced background job processing delays for 2.2 hours, affecting the Background Processing component. The engineering team identified the root cause of the delays and worked on implementing mitigating measures to resolve the issue.

A configuration update caused intermittent error messages on GitLab Next (Canary) for 1.7 hours. Users were advised to use GitLab Current as a workaround while the fix was implemented. The team identified the root cause and worked on resolving the configuration issue.

GitLab experienced resource issues with their small hosted runners (saas-linux-small-amd64) that caused delays for CI/CD jobs tagged for these runners. The incident affected GitLab SaaS Shared Runners, creating a backlog of pending jobs that users experienced as processing delays. The team implemented configuration changes to improve resourcing and resolved the issue after 1.1 hours.

GitLab experienced a parsing error in vulnerability reports that affected dependency scanning functionality on their website for 2.2 hours. The team identified the root cause of the issue and worked to resolve it. The incident was classified as minor severity and has been resolved.

GitLab experienced degraded performance in their AI-assisted services and AI gateway, causing users to encounter higher latency and errors when using Duo features. The incident lasted 4.6 hours and was caused by request saturation and potential segfaults affecting the AI services. GitLab resolved the issue by increasing maximum concurrency limits to reduce latency.