Outage in Technolutions

Overnight background services delayed

Resolved Minor
February 06, 2024 - Started over 1 year ago
Official incident page

Incident Report

An AWS-initiated overnight update to a series of Redis servers used for background service queue management has delayed background queue processing as the queues entered a stalled state. Delayed queues are processing now and will complete shortly.

Need to monitor Technolutions outages?

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Try IsDown risk-free 14-day free trial · No credit card required
Latest Updates ( sorted recent to last )
RESOLVED over 1 year ago - at 02/08/2024 12:57AM

No issues have been observed since the overnight background service queue delays on 2/6, and all overnight queues on 2/7 performed as expected. We will continue to monitor background services to ensure ongoing operation within desired ranges.

MONITORING over 1 year ago - at 02/06/2024 10:14PM

As of earlier this afternoon, fewer than 1% of databases had ongoing overnight jobs still executing. The remaining databases have jobs that typically take hours overnight to complete, and with the accumulation of daytime jobs and blocking processes on their databases, are continuing to run. Separately, we have adjusted the background process monitoring to better recover from a scenario in which the Redis servers become temporarily unavailable due to hardware or software updates to those nodes.

MONITORING over 1 year ago - at 02/06/2024 01:25PM

An AWS-initiated overnight update to a series of Redis servers used for background service queue management has delayed background queue processing as the queues entered a stalled state. Delayed queues are processing now and will complete shortly.

The Status Page Aggregator Built for IT Teams

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4522 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook