500 Errors due to failures in Gitaly service nodes

We see a 1.5% degraded state on two of our Gitaly nodes. Some customer may experience delays while contributing to their projects. We have identified the problem and we are working through steps to resolve this.

https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

Partial service disruption due to this issue.

We have recovered and are still monitoring.

https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

We have seen an unexpected increase in 500 errors on GitLab.com pages, often related to Merge Requests. We are investigating the root cause of these issues.

Further details in the incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

We are continuing to investigate the 500 errors on GitLab.com. We are investigating the root cause of the incident. We are preparing to apply mitigations to reduce error rates.

Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

We have rebooted the VMs that were generating 500 errors. Once the VMs have rebooted we will validate that Gitaly is responding normally and work to identify the root cause of these issues to prevent re-occurrence.

Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

The nodes have been rebooted and Gitaly appears to be working as expected. We are moving this incident to Monitoring. Our team is working to identify the root cause of these errors while we monitor the node performance.

Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

Gitaly appears to be working normally at this time. We are continuing to monitor the issue and have mitigation steps should the issue reoccur. We do not anticipate further updates unless we start seeing degraded Gitaly performance again.

Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

We're observing elevated 500 error rate on GitLab.com again and we're investigating the root cause.

See detail issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

We've identified the problem related to git activities. We're looking into solution.

See issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303