GitLab Outage History

GitLab experienced degraded performance in their AI-assisted services and AI gateway, causing users to encounter higher latency and errors when using Duo features. The incident lasted 4.6 hours and was caused by request saturation and potential segfaults affecting the AI services. GitLab resolved the issue by increasing maximum concurrency limits to reduce latency.

GitLab experienced CI pipeline processing delays for 2 hours, where jobs were not being picked up by runners or remained stuck in pending status. The issue was caused by infrastructure capacity errors that prevented proper job execution. The incident was resolved by implementing an infrastructure change that improved pending job metrics.

GitLab experienced failures with runners pulling container images due to a problematic merge request (MR). The issue affected CI/CD pipelines that relied on container image pulls during job execution. The incident was resolved by reverting the problematic change after 1.7 hours.

GitLab.com experienced a database performance issue that caused slower than normal response times across the website and API for 1.6 hours. The degraded database query performance affected user interactions with GitLab's web interface and API endpoints. Performance was restored after the database issue was resolved, though monitoring continued while the underlying cause was investigated.

GitLab Duo services, including Agentic Chat and Code Suggestions, experienced degraded performance for 11.5 hours due to an upstream provider outage affecting Claude models. The incident caused intermittent issues with GitLab's AI-powered features on their website. Services returned to normal operation once the upstream provider resolved their outage.

GitLab experienced intermittent Git clone failures and CI/CD pipeline timeouts lasting 11.9 hours, with users encountering load-related errors and Gitaly timeout messages. The incident affected Git Operations and was initially addressed by rolling out a feature flag, which was later rolled back in favor of exploring alternative long-term solutions. The issue was resolved with error rates returning to low levels after the feature flag rollback, though monitoring continued to ensure stability.

GitLab.com experienced login failures for users attempting to authenticate with email-based two-factor authentication, with the issue confirmed to affect only free-tier users. The incident lasted 12.3 hours while the team investigated logs and identified patterns in sign-in endpoint requests. The issue was mitigated and users can now successfully log in with email-based 2FA, though root cause investigation continues.

GitLab's Duo Agent Platform and LLM proxy endpoints experienced rate limiting issues that caused errors when users attempted to access Claude models for 12.4 hours. The incident was resolved by implementing traffic filtering rules to block excessive requests, which reduced rate limit errors to near zero and restored Claude model availability. A permanent fix was deployed with continued monitoring for follow-up improvements.

GitLab's gitlab-org repository experienced 500 errors for 44 minutes due to a faulty Gitaly node, preventing users from accessing the repository through pipelines and other tasks. The service has been restored and users can now access the gitlab-org namespace while integrity checks are being performed to ensure full recovery.

GitLab experienced delays in background job processing for 4.4 hours, causing users to see delays in CI job status updates. The team investigated the issue and developed improvements to address the processing delays. The incident was resolved after implementing fixes that were under review during the outage.