GitLab Outage History

A code change caused 500 errors across GitLab's REST and GraphQL API endpoints for 9.4 hours. GitLab identified the problematic code change and reverted it, with the fix fully deployed to production by the end of the European business day. Users experienced API failures until the revert was completely rolled out, with a temporary workaround available through the Canary environment.

GitLab experienced a 4.2-hour delay in background job processing that caused CI job status updates to appear slowly in the user interface. The engineering team identified and deployed a primary fix, with full rollout taking several hours and additional mitigations implemented in parallel. The incident was resolved after the fix was fully deployed across the system.

GitLab experienced CI pipeline processing delays for 1 hour due to background job processing issues, with small CI runners being primarily affected by increased job queuing rates. The incident was caused by a Sidekiq backlog that led to CI job accumulation and subsequent delays. The team observed recovery by the end of the incident period while continuing to investigate the root cause.

GitLab experienced intermittent errors affecting Duo Workflow and AI Gateway for requests using GPT models, with users also encountering NotImplementedError issues in Code Review flows. The engineering team identified the root cause as a recent commit and worked to revert the problematic changes. The incident lasted 2.9 hours and was resolved through a deployment that reverted the recent changes.

GitLab experienced widespread 5XX errors affecting their website, API, and Git operations for 16 minutes. The incident was classified as major due to the broad impact across core services. The cause was under investigation during the outage period.

GitLab users experienced 403 Forbidden errors when attempting to download NuGet packages starting at approximately 08:00 UTC on March 10, 2026, affecting both the website and API. The issue was caused by a problematic merge request that required reverting. The incident lasted 13.4 hours before resolution.

GitLab experienced a 2.5-hour incident where CI/CD job processing was delayed due to PgBouncer saturation affecting background job processing. The team identified the root cause as database connection pooling issues and implemented mitigation efforts. The service stabilized and recovered, with the team continuing to monitor for full resolution.

GitLab experienced CI/CD pipeline processing delays lasting 2.3 hours due to a Sidekiq backlog and resource issues. The engineering team resolved the incident by disabling a specific worker and deploying configuration updates to clear the backlog. Normal pipeline processing was restored after the main backlog drivers were mitigated.

GitLab CI/CD pipelines remained stuck in "Running" status as a lingering effect from a previous incident on March 5th. The issue affected pipeline execution and prevented proper completion of automated builds and deployments. GitLab resolved the incident by developing and implementing a process to identify and safely correct the stuck pipelines over a 2.8-hour period.

GitLab experienced errors with pulling from the main repository, affecting Git operations for 34 minutes. The issue impacted users' ability to perform standard Git pull operations on the main GitLab repository. The service disruption was resolved after investigation and remediation efforts.