Outage in Buildkite

Test Analytics availability

Resolved Major
September 06, 2022 - Started over 1 year ago - Lasted about 7 hours
Official incident page

Need to monitor Buildkite outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Buildkite, and never miss an outage again.
Start Free Trial

Outage Details

We're investigating some database issues that are currently impacting availability of Test Analytics.
Latest Updates ( sorted recent to last )
MONITORING over 1 year ago - at 09/06/2022 10:57AM

The restored Test Analytics database is now operational. We’ve restored service availability and are now ingesting new executions. We will continue working with AWS to attempt to restore test executions ingested from 01:17 UTC through 03:55. Test executions that were attempted to be sent to Test Analytics between 03:55 and 10:32 were not ingested and are not recoverable.

IDENTIFIED over 1 year ago - at 09/06/2022 10:11AM

We have restored our database onto a new AWS Aurora cluster and are attempting to restore the connection.

IDENTIFIED over 1 year ago - at 09/06/2022 09:18AM

We are still working with our upstream provider on the Test Analytics database recovery.

IDENTIFIED over 1 year ago - at 09/06/2022 08:32AM

We are continuing to work with our upstream provider on the Test Analytics database recovery.

IDENTIFIED over 1 year ago - at 09/06/2022 07:57AM

We are continuing to work with our upstream provider around our database recovery.
We have also shipped a fix for team administration in the Buildkite UI.

INVESTIGATING over 1 year ago - at 09/06/2022 07:21AM

We are still working with our upstream provider, around a point-in-time recovery as a contingency. We are still working on remediating team administration.

INVESTIGATING over 1 year ago - at 09/06/2022 06:40AM

Our upstream provider continues to investigate, and the point-in-time recovery contingency option is nearly ready, should we need it

INVESTIGATING over 1 year ago - at 09/06/2022 06:06AM

While we are investigating the problem with our upstream provider, we are preparing a point-in-time recovery as a contingency. We have identified that administering team permissions are also impacted.

INVESTIGATING over 1 year ago - at 09/06/2022 05:29AM

We are continuing to investigate this issue with our upstream provider.

INVESTIGATING over 1 year ago - at 09/06/2022 05:11AM

Availability of Test Analytics is currently impacted. We have escalated to our upstream provider and are continuing to investigate the cause of the problem.

INVESTIGATING over 1 year ago - at 09/06/2022 04:29AM

We're investigating some database issues that are currently impacting availability of Test Analytics.

Latest Buildkite outages

Delayed notifications - 22 days ago
Delayed dispatch - 28 days ago
Degraded Performance - 2 months ago
Degraded Notification - 3 months ago

Start monitoring Buildkite and all your cloud vendors in minutes

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3153 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime