Sync servers crashing

Systems are operational, no incidents since last update, we don't expect further downtime while we optimize the data center.

Earlier today we identified an issue where ~9% of branches experienced extra delay on both syncs and deploys. The fixes from yesterday helped containing the issue so the other 91% of branches could continue as normal. We've resolved this issue by scaling up the units that have to do the work. There are currently no known issues, but we are looking for a sustainable solution, which may involve one or more releases.

In technical terms: The scaling and self-repair configuration we've used for the last year is no long viable. We've greatly optimized on CPU, to the point where it is no longer a reliable metric for scaling in or out. We've disabled the scaling, and manually put the system on a configuration that works right now, and continue to fine tune it while we work on a permanent and automated solution.

The "ring" that serves syncs and deploys is stable again. This should get the syncs and deploys running as usual. In the meantime we'll keep an eye out and act immediately if something changes.

Unfortunately the system degraded again after ~1 hour.

We have identified the cause of the intermittent delays affecting branch synchronization and related processing over the past day. A fix has been deployed and the affected processing infrastructure has been replaced. All synchronization services are operating normally and we are monitoring closely to confirm the fix holds under full load.

Synchronization jobs that were delayed during this period have resumed automatically — no action is required from customers, and no data was lost.

Syncs and deploys are currently not happening. The underlying servers are crashing for various reasons. We're working on pinpointing the exact issue.