ShowClix and Ticketbooth Outage 5/26/25

At 8:20 PM EDT replication stopped on a read-only replica database which caused 404 responses when accessing data created or updated after that time. Engineering attempted to restart the database at 10:05 PM. However the process did not restart on its own, leading to connection failures to the replica database for a period of around 6 minutes. At 10:11 PM DEVOPS pulled the affected replica out of rotation which resolved all issues. Practically the effects were that customers and clients experienced “not found” errors when viewing some data (viewing orders, event configurations, etc.) between 8:20PM and 10:05 PM and widespread 500 errors (which render as the broken ticket page) when accessing the application (purchasing tickets, logging in, accessing the admin, etc). between 10:05 and 10:11 PM. This affected all ShowClix, Ticketbooth and white label clients.