Timeouts when interacting with upstream GitHub mirrors and GHCR

We are currently seeing signs of upstream degradation causing some interactions with GitHub such as checkouts or pushes and pulls to GHCR to timeout.

We are exploring a proxy based solution to circumvent the hops in our network that are resulting in degraded interactions with GitHub. We apologize for the extended nature of this outage and request you to reach out through our support portal in `https://app.blacksmith.sh/?support=open` if your org is currently blocked so that we can provide you with a workaround.

We are continuing to look into this issue, it seems localized to one region and we are working on identifying the source of the degradation. We will follow up with an update as soon as we know more.

Our network provider has implemented a fix to remove an ISP that was affecting return traffic from GitHub's edge nodes. We are seeing recovery in our US region and will continue to monitor the situation. We sincerely apologize for the disruption in service. A detailed postmortem will be shared about what happened here along with what we're doing to prevent such an extended outage in the future.

We're working with our network provider to understand why 10-15% of request failure are seeing 5+ second latency spikes to GitHub services. We're working on identifying if this continues to be an issue upstream or in our network stack.

We realize this is affecting several customer workflows, we are actively working on a resolution.

Some evidence suggests that there's an issue with the upstream GitHub edge node that we're hitting from our US region, but the location being hit from our EU region doesn't have the same degradation. We're still monitoring.