Increased error rate affecting Web UI and APIs

This incident has been resolved. All services are operating normally and remaining jobs have been processed.

We apologise for the disruption and thank you for your patience.

All services are currently stable and we continue to monitor system status. We are working to unblock some remaining jobs that were impacted by the incident.

Thank you for your continued patience.

The issue has been resolved and service has been restored to all remaining customers. We are continuing to monitor to ensure stability.

We apologize for the disruption and thank you for your patience.

We have restored service for additional customers and are continuing to work on a resolution for remaining affected customers.

We've identified the root cause of the issue: a database index on our annotations table was replaced, but the replacement index isn't being utilized as expected under load, which is causing performance degradation.

Thank you for your understanding and patience as we work to resolve this as quickly as possible.

We have restored service for additional customers and are continuing to work on a resolution for remaining affected customers.

We are actively working on a resolution for remaining customers. We do not have an estimated time for resolution at this time.

We have restored service for the majority of customers and are continuing to work on those that remain.

We are working through applying a fix but the exact time to fully restored service is not yet clear.

The REST and GraphQL APIs, as well as the Web UI, are currently affected by increased error rates and slow performance.

We have identified the cause of the issue, and are working to address it.

We have noticed issues with displaying build annotations, which may cause viewing builds in the web UI. We are still investigating and will provide another update shortly.