Investigating Service Disruption

This incident has been resolved.

> What happened
Following a planned infrastructure update last night, some customers experienced errors when performing certain actions on the platform, particularly around candidate pages and interview scheduling. We identified the root cause and restored full service by reverting to our previous database infrastructure.

> Current status
The platform is now fully operational. All features are working normally.

> Data impact
Due to an issue between both database providers when we restored service, there's a short window (between 10:16 and 10:57 UTC this morning) where some changes made on the platform may not yet be reflected. No data has been lost, we have complete records of everything that happened during this window and are working to restore any affected changes to the live system.

The status of data reconciliation is being tracked here: https://status.pinpoint.support/incidents/9v65gddb3y1n

> A note on our recovery standards
We maintain internal Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) as part of our commitment to platform reliability. The 41-minute data gap from this incident falls outside the standards we set for ourselves, and we take that seriously. While no data was lost, we're treating this as a priority learning for our infrastructure processes going forward, and feedback for our vendors and their practices.

A full post-mortem will follow on Monday.

We are currently experiencing platform issues affecting candidate index pages as well as some actions across the platform including editing or deleting certain types of records (e.g cancelling and re-scheduling candidate interviews). This follows a planned infrastructure update completed last night.

We’ve identified some unexpected issues related to the change and are implementing adjustments to ensure full stability. This work involves a few coordinated steps and is expected to take around 30 minutes.

We’re treating this with the highest priority and will continue to provide updates. Thank you for your patience, and we apologize for the disruption.