Increased latency for large repos

This incident has been resolved but we believe it triggered a worsened incident overnight Sun night/Mon morning (US PDT) which has just been resolved. To be confirmed by full RCA, we believe a deluge of large repo uploads tied up individual web servers that handle frontline requests. Each web server is able to recover on its own, and did, but as volume increased all servers were eventually affected, only allowing short windows where requests could get through—rejecting most requests with 504 errors.

We'll post a post-mortem when we understand more about what happened and how to prevent it going forward.

We are continuing to monitor this situation. Latency is much reduced but still elevated. We will close this incident when it's fully restored to normal.

We are monitoring increased latency for larger repos (5K+ source files) due to elevated traffic in background job queues. We will pause outlier repos (more than 2x average traffic) to allow queues to clear for the general population, then restore once queues have cleared. If you think your repo may have been one of those paused, two things:

1) Reach out to us at support@coveralls.io and we'll confirm; and
2) Consider purchasing a usage add-on so that your jobs are not throttled for fair-use, or an enterprise plan with isolated infrastructure for maximum performance: https://coveralls.io/pricing