Bitrise Outage History

Bitrise experienced elevated error rates affecting Android builds due to issues with their Maven Central mirror implementation, including edge cases where the mirror couldn't be applied and compatibility problems with dependency verification. The incident lasted 34.3 hours and impacted the web app component. The issue was resolved after teams identified the root cause, released fixes for the affected steps (Install Android SDK components and Activate Gradle mirrors), enabled mirror redirection by default for all builds, and addressed upstream connectivity problems.

Bitrise experienced elevated error rates for Android builds due to Maven Central implementing rate limiting that caused 429 errors when downloading dependencies. Bitrise responded by implementing a proxy layer to mirror Maven packages and created a dedicated "activate-gradle-mirrors" step to help customers bypass the rate limiting issues. The incident was resolved after 9.8 hours by distributing traffic flow evenly and addressing edge cases in customer Gradle configurations that were bypassing the mirror.

Bitrise experienced elevated error rates affecting Android builds for a subset of users over 8.6 hours, initially caused by MavenCentral request throttling and later identified as issues with the Gradle plugin portal not being found in the "Install missing Android SDK components" step. The engineering team implemented redirections and fixes to resolve the throttling and plugin portal issues, while recommending users add the "Install missing Android SDK components" step as a mitigation during the incident. The incident was fully resolved with all impacted services restored to normal operation.

Bitrise experienced elevated error rates that caused build details pages on app.bitrise.io to fail loading for a subset of users, with HTTP errors, intermittent page load failures, and timeouts occurring. The engineering team released a fix to mitigate the issue and monitored systems until the incident was fully resolved after 5 hours.

Bitrise experienced a billing system issue where credits and builds were not being refreshed at the start of billing cycles for a subset of users. The incident also affected account, team, and billing management features on the website. The issue was identified and resolved after 3.2 hours of investigation and remediation.

Bitrise experienced elevated build queue times affecting standard, elite, and elite-xl Linux machines, causing users to wait longer than usual for builds to start. Build execution remained normal once workers were allocated, but the queuing process was significantly delayed. The engineering team identified the root cause and deployed a fix, resolving the incident after 32 minutes.

Bitrise experienced elevated queue times on Linux machines that affected the Web App for 5.9 hours. The issue was identified and a fix was implemented. The incident has been resolved.

Bitrise experienced build-related issues affecting their web UI, preventing users from starting new builds, processing incoming webhooks, and accessing build artifacts of running builds. The problems were identified as stemming from a Google Kubernetes Engine incident, and Google applied a workaround that restored normal operations. The incident was fully resolved after 3.4 hours.

The Deploy to Bitrise step was failing due to network issues, affecting the Web App component for 33 minutes. The issue was identified and a fix was implemented, with the incident fully resolved after a brief monitoring period.

Bitrise experienced increased build queues across all stacks, affecting the Web App for 1.6 hours. A fix was implemented and monitored before the incident was resolved. The issue impacted build processing times for users across the platform.