Queue Processing Delays

On the 20th of Jun 2024, Tanda experienced severe degradation in our background Queue processes. These processes are used to update Timesheets with clockins, pull data from integrations (i.e PoS Systems), send notifications and various other background tasks. This incident started at 12pm AEST on the 20th of June with minor performance problems, and at 4pm AEST escalated into a larger degradation of performance across all queues. The team worked to resolve this issue until 12am AEST on the 21st of June with queue processing resuming normal behaviour. All clockins sent in this time have been processed, and no data was lost. During this time, the website and API was fully operational.