Technolutions Outage History

Technolutions experienced a 1.7-hour outage affecting the availability of certain databases used by their Slate service. The interruption was caused by a loss of quorum in the failover cluster, which was linked to periodic network issues following recent database infrastructure and operating system upgrades. The team resolved the immediate issue by bringing the databases back online and hardened the cluster configuration to prevent future unintended failovers.

Databases on LUNA experienced intermittent availability for 3.6 hours due to a high availability service that initiated an unexpected failover and became stuck in a failover loop, potentially related to weekend database server upgrades. The Slate component was affected during this incident. The issue was resolved by temporarily disabling automatic failover while investigating the root cause, with manual failovers implemented as a protective measure.

Technolutions experienced intermittent availability of certain databases in the US region for 4.6 hours, primarily affecting databases on the LIMA cluster and the Slate component. The issue was caused by a routine third-party database engine update that introduced a behavioral change leading to worker thread exhaustion, which triggered an automated failover to secondary infrastructure where some databases were slow to recover due to similar thread exhaustion. The incident was resolved by disabling the problematic functionality as advised by the vendor and increasing the worker thread ceiling to prevent recurrence.