Processing delays for repos with 5K+ source files

This incident has been resolved.

Continuing to monitor. Build times coming down for large repos. Fewer than five (5) repos now affected.

Build times have come down for large repos. We are continuing to monitor as associated queues clear.

We've seen a spike in build times for large repos (5K+ source files). Right now, only nine (9) repos seem to be affected. We have implemented a fix and are monitoring as build times return to normal for these repos.

The root cause of this spike is elevated jobs (and therefore dequeue times) in queues dedicated to large repos which appears to have been triggered by two (2) repos sending an unusually high number of uploads over the past 1-3 hr period. We have increased resources to these queues to accelerate processing but may need to pause the above repos if uploads continue at the same level.

If you think your repo was affected you can reach out to us at support@coveralls.io. We will reach out to the owners of the repos with elevated jobs.