Elevated Latency

We’re now closing this incident, several hours after restoring full system stability. Over the past 4 hours, we’ve continued to monitor key requests and queries closely. During that time, we identified a number of previously long-running queries that we’ve either:

- Optimized immediately, based on new platform characteristics; or
- Added to a short-term optimization backlog for tuning over the next few days.

These efforts are part of our ongoing work to adapt all app queries to the updated infrastructure context.

Site remains fully operational and performance for all new builds is normal; but we will continue monitoring request times and individual query times to identify long-running queries that may have caused recent delayed job backups / spikes in latency.

We believe performance has been restored, and site is fully operational, but we will continue to clear any previously blocked (or retry) jobs we discover in background job queues and monitor performance stats as they clear.

Monitoring for further issues. Performance for new builds is normal. Waiting for dequeue metrics to fall below 50% normal before we lift "degraded performance" rating.

We believe the issue is resolved. We are scaling infrastructure to clear any delayed background jobs, and monitoring to ensure latency stays within normal range.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are investigating elevated latency in our background jobs system. Some users have also reported receiving Timeout errors while trying to load web pages.