Semaphore experienced slow communication between jobs and GitHub, causing delays when checking out repositories via SSH. The issue affected Semaphore 2.0 operations for 6.8 hours, with users needing to use a retry command workaround during the incident. The problem was resolved by deploying a network fix to re-route GitHub operations to healthier servers.
Trusted by 1,000+ teams
Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.
This incident has been resolved.
A fix has been implemented, and everything should be back to normal. The previously shared snippets can be removed.
We identified slow GitHub operations related to problems with the network, and we deployed a fix to re-route those operations to a healthier server. Some HTTPS operations started to fail because of that. To disable the fix, you should use the command:
sudo sed -i 's/^\([0-9].*github\.com\)/#\1/' /etc/hosts
And to re-enable, you can use:
sudo sed -i 's/^#\([0-9].*github\.com\)/\1/' /etc/hosts
A fix has been implemented and we are monitoring the results.
We identified slow communication between Semaphore jobs and GitHub. We are continuing to work with our upstream provider to resolve this. For impacted jobs, please replace the normal checkout command with the following:
retry --times 5 "timeout 40 bash -lc 'source ~/.toolbox/toolbox; checkout'"
And afterwards, use cd to change the directory to the cloned repository (replace with your project).
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 6320 services available
Integrations with