Spike in Check API 5xx Errors

Between approximately 19:11 and 19:23 UTC, a subset of Check API requests returned 5xx errors, which also intermittently impacted Console and Components. The errors occurred because a routine deployment introduced a new data value that application instances still running the previous version did not yet recognize; while the rollout was in progress, requests served by those older instances failed. Error rates returned to normal at 19:23 UTC once the deployment fully completed and all traffic was handled by the updated version.

We are implementing safeguards, including updated data-migration and deployment practices plus additional automated review checks, to prevent similar issues in the future.

No customer action is required, and the issue did not affect payment or payroll processing.

The system has recovered and is now operating normally.

The issue was transient and related to an in-progress deployment. We are continuing to monitor the system closely and are conducting a root cause analysis to better understand the issue and prevent similar occurrences in the future.

We have identified an increased number of 5xx errors in Check API, which is also impacting Console usage. Investigation is in progress.