Intermittent availability of certain databases

There have been no continued impacts.

The incident, which primarily affected databases on the LIMA cluster, was caused by an automated failover to secondary infrastructure. A routine servicing update to the third-party database engine introduced a behavioral change that, under specific conditions, resulted in an exhaustion of worker threads. Upon failing over, some databases were slower to recover as a result of a related exhaustion of worker threads on this secondary node. We have disabled the functionality that resulted in this behavioral change, as advised by the vendor, and have increased the ceiling for worker threads to reduce the potential for reoccurrence.

Everything remains stable at this time, and we will continue to monitor for any further issues.

We are continuing to monitor for any further issues.

All affected databases recovered approximately a half-hour ago. We're continuing to address some internal connectivity issues, but we're not seeing any observable impacts at this time. There may be brief connection interruptions as failbacks complete later.

We are investigating the intermittent availability of certain databases in the US region. Some databases are still recovering following a failover to secondary infrastructure and may be unavailable during the completion of the failover. We will provide updates shortly.