Hanging status updates

All queues are cleared. As a result, all previously reported delayed builds and or status updates should now be complete / received.

We are not seeing any further backups in any queues, but will continue monitoring into morning when our usage increases.

If you are still experiencing any unfinished builds, or delayed status updates, please reach out and let us know at support@coveralls.io.

All backed up queues are fully drained. There is now a flurry of activity in some associated queues, which are completing the processing and notifications of previously delayed builds, but those are processing quickly and we expect any and all builds and notifications previously reported as delayed today to be finished and complete in the next 30-45 minutes.

Our fix has been fully deployed and we will be monitoring for any further backups.

The backed up queue affecting all users has drained by 75%. Our fix is still being deployed across all servers, but should start taking effect in the next 15-20 min.

We have scaled up processes on clogged background queues and they are draining. We have also implemented a fix we hope will avoid further backups and are monitoring for effects.

We have identified the root cause of delayed status updates for some repos (reported today) as backups in several queues that process background jobs pertaining to aggregate coverage calculations for new builds, which precede the sending of notifications and are therefore delaying those.

However, we have not yet identified a pattern behind these spikes or the delays in processing these queues since none of our usual performance metrics had been triggered (until recently when a queue that affects all users triggered an alarm).

We are scaling up server processes to clear that backup, but since we are not seeing degraded performance metrics from servers, we are continuing to investigate other causes for delayed processing.

Several customers have reported long delays receiving status updates for new builds at GitHub, or status updates that have hung and never arrived. We are investigating the issue.

If you are experiencing this issue, please reach out and let us know at support@coveralls.io so we can include your cases in our investigation.

Note that there were some incidents receiving API requests at GitHub in the last 24 hrs, per this status update from GitHub:
https://www.githubstatus.com/incidents/gqj5jrvzjb5h

We evaluating cases against this timeframe to understand if they align with the GitHub incident period.