Incident with Pull Requests, Git Operations, Actions, API Requests, Issues and Webhooks

This incident has been resolved.

Issues is operating normally.

Actions is operating normally.

Pull Requests is operating normally.

Webhooks is operating normally.

Git Operations is operating normally.

API Requests is operating normally.

We are seeing site-wide recovery but continue to closely monitor our systems and putting additional mitigations in place to ensure we are back to full health.

We are continuing to see consistent impact, and we’re continuing to work on multiple mitigations to reduce load on our systems.

We have found an issue that may be contributing additional load to the web site and are working on mitigations. We don't see any additional impact at this time and will provide another update within an hour if we see improvements or fully mitigate the issue based on this investigation.

We have taken some mitigations and see less than 0.3 percent of requests failing site wide but we still see elevated 500 errors and will continue to stay statused and investigate until we are confident we have restored our error rate to base line.

We are seeing increased 500 errors for various GraphQL and REST APIs related to database issues. Some users may see periodic 500 errors. The team is looking into the problematic queries and mitigations now.

Actions is experiencing degraded performance. We are continuing to investigate.

Git Operations is experiencing degraded performance. We are continuing to investigate.

Pull Requests is experiencing degraded performance. We are continuing to investigate.

Webhooks is experiencing degraded performance. We are continuing to investigate.

Issues is experiencing degraded performance. We are continuing to investigate.

We are investigating reports of degraded performance for API Requests