Elevated error rate with Android builds

This incident is now resolved, and impacted services are now operating.

Our teams identified the root cause, released a fix, and were monitoring our systems continuously to ensure the mitigation is safe. We'll conduct an internal post-mortem to understand what happened and learn how to avoid a similar outage in the future.

Thank you for your patience and understanding. If you encounter any issues, please reach out to our Support team:

https://support.bitrise.io/en/

We are continuing to fine-tune our mirror for reduced latency. Customers using our mirror are no longer experiencing rate limiting. We are continuing to monitor for edge cases and help customers migrate to using the mirror by using the following guide.


When will the next update be available?

You can expect a status update approximately 1 hour from now.

What we’re doing?

We have rolled out a change so that all builds have the redirection enabled by default; there is no need to add the specific steps for this, except for dependency verification (which requires a specific step version pin).

We identified upstream 401 errors and are working on a fix.

We identified a gap in the mirroring logic for Roboelectric tests, exploring our options to help mitigate the issue.

We are seeing larger latencies from Maven Central upstream, which we are investigating.

When will the next update be available?

You can expect a status update approximately 1 hour from now.

For customers having dependency verification enabled for their Android builds, you will need to update your configuration based on our updated guide.

We have identified cases where customers have cached the init scripts performing the mirror redirection. This can break the builds as the mirrors are deployed inside the datacenters, so one URL in a build won't work in another. We strongly suggest that customers to not cache anything in ~/.gradle/init.d/, and if you do, clear these contents / re-add our steps that activate the mirror.

When will the next update be available?

You can expect a status update approximately 1 hour from now.

We are seeing some timeouts when connecting to the Maven Central proxy in one of our datacenters and are investigating. Dependency verification support is still in progress. We are exploring options to auto-enable the Gradle mirrors for our Bitrise customers.

When will the next update be available?

You can expect a status update approximately 1 hour from now.

We are tackling edge cases where the Maven Central mirror cannot be applied. We are working on a fix to make the Maven Central mirror compatible with dependency verification. We have released new versions of the affected steps (Install Android SDK components and Activate Gradle mirrors) to cover a larger subset of artifacts to proxy

Services impacted:
Android builds

Severity:
SEV-2

When will the next update be available?

You can expect a status update approximately 1 hour from now.