High Error Rate for GitLab.com projects API

We are currently seeing a high rate of errors for the "/api/v4/ /projects" endpoint on GitLab.com. Our infrastructure team is working on identifying the root cause.

For more details, please refer to the following issue:
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7516

No substantial updates at the moment, the investigation is still ongoing.

While we're continuing to investigate the root cause, we might've identified a mitigation: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1989

The potential mitigation is currently being propagated throughout our infrastructure, and we’re measuring the impact.

The mitigation is in place and we’re no longer being able to reproduce the issue. The error rates for the projects API endpoints have recovered.

We will continue to monitor the error rates while we investigate the root cause.

Our infrastructure team identified the root cause: Rails version 6.1.5.1 introduced CSP headers on non-HTML responses, which caused our load balancers to drop responses.
We’re currently investigating other endpoints: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7516