Extended period of degraded performance for some customers - 17th Feb 2022

From 11:50 AM to 1:50 PM on Thursday 17th Feb, one group of aXcelerate customers experienced an extended period of degraded system performance with two brief outages (impacting one in eight customers in production). Average page load times were between 5 and 10 seconds during this period. At 12:43 AM and 1:28 PM there were outages for approximately two minutes in both instance. The issue was caused due to higher than expected traffic on these servers. In response, aXcelerate has increased the number of servers in the pool that supports these clients. We have also improved the alerts and notifications that will ensure we are able to respond to these types of events faster in future. As an ongoing activity, aXcelerate engineers will be working to improve the load balancer configuration that will allow servers to automatically scale up when traffic is higher than usual.