Test Engine services unavailable

We believe this issue is now resolved and all Test Engine systems are functioning. Ingestion of test data was delayed during this incident but we believe there has been no data loss.

Customers using Buildkite Test Engine Client (bktec) to run tests will have experienced slower builds during the incident due to bktec using a sub-optimal test planning strategy while Test Engine APIs were unavailable.

Our third party database provider has recovered their systems and we are have worked through our backlog of test executions. We are continuing to monitor to ensure our systems are healthy.

We have been notified by the third party provider that they are currently restoring services. We will update when we have more information.

A third party database provider is experiencing an outage which is resulting in Test Engine services being unavailable. We will report updates as we receive them.