Chameleon application errors / performance

This incident has been resolved.

Response times and error rates are returning to normal and will continue to monitor. At this time, we know that a large increase in latency when writing to the database caused the latency of essentially all other operations to increase.

We will leave this incident open until we're sure that the changes have a chance to full stabilize, the background jobs have processed, and the workload returns to normal levels/latencies.

We are seeing elevated error rates, and various systems being slow or not loading. At the moment this includes the Chameleon dashboard, identifying users, and sending data to Chameleon. We know this is related to the performance of our primary database and will have an update again shortly.