Order Service Failures

On Tuesday afternoon we had two spikes in database query response times. After further investigation we have attributed these to query wait time caused by table locks. These locks caused connection saturation which caused API request timeouts. We have changed the write ahead logging size on our database to mitigate this issue. We will continue to monitor over the next couple days while we add alerting to monitor for this scenario in the future.

Everything is functioning as expected, and we have an active P1 case open with our cloud vendor to discover the cause of yesterday's 2 issues.

We have seen multiple short-term slowdown events for Order Service. We are investigating this issue and will share any updates here. We apologize for any inconvenience this may have caused.