Test Insight Results are Delayed

The test results processing pipelines is starting to recover. We are caught up to builds that ran within the last 15 hours and expect to be fully caught up in 2-4 hours.

We have deployed both changes to speed up test result processing.
We are now deploying changes to remove unprocessable events from the queues.

The issue with the Insights web UI has been resolved.
Engineers are also monitoring metrics for the fix to test insights processing, once things have stabilized we will complete the rollout of the second update, at which point we believe that processing will return to normal, and we will begin to work through the backlog.

Due to an unrelated issue with a back-end database, the Insights UI is currently unavailable. Engineers are currently working on remediation.

We are working on the second of two updates at which time we expect to see accelerated processing of the backlog

Insights data for tests (flaky tests, slowest tests, most failed, etc) is currently delayed by approximately 2 days.
Historical data remains visible and test data has not been lost.
We are in the process of deploying multiple changes to accelerate processing and will update as we start to see the delay time reduce.

We are continuing to work on a fix for this issue.

We have identified an issue with Insights (affected may notice that the data shown for flaky tests, slowest tests, most failed tests, etc. are delayed). A remediation is currently in progress