Issues with slow processing of results

We are continuing work on full recovery. Initial results processing and notifications are back to normal, but some data queues that populate the dashboard charts and explorers are still delayed.

We are almost fully recovered, though are still seeing a 10 minute delay on our analytics queues, that power some of the reports in the dashboard. These should finish catching up over the next half hour, with the delay improving steadily.

We are currently investigating this incident. But processing results has gotten significantly behind. We are also seeing an increase of errors in the endpoints that receive client requests from the reports.

We are continuing to work on recovery. The endpoint errors have been resolved. But our queues are backed up, so results are taking a while to show in the dashboard.