System Performance Degradation

Summary:
Between 10:20am - 10:45am and 12:35pm-12:50pm some of our web servers experienced an application process hang, causing health probe failures and degraded performance for a subset of requests. The affected servers were removed from the pool, and traffic continued to be served by the remaining healthy instances.

Impact:
Some users may have experienced slow response times or brief connection delays during the incident window

Resolution:
An additional server was provisioned to provide extra capacity while the affected servers were restored to the load balancer pool. All servers are now operating normally.

Next Steps:
We are continuing to investigate the root cause, which appears related to memory pressure during a period of high concurrent activity. Additional monitoring has been implemented, and we will provide a full post-incident report once the investigation is complete.

Current Status:
All systems operating normally.

CPU usage has remained stable across all servers over a sustained period. We will continue to monitor closely and provide further updates as needed.

We are observing a CPU usage spike on a different server and are continuing to monitor the situation closely. Further updates will be provided as they become available.

CPU usage has stabalised on our web front-end servers and performance has returned to normal. We are continuing to monitor closely.

We have identified the affected server and removed it from the pool to reduce impact while we continue to investigate the root cause. Further updates will be provided as available.

We are continuing to investigate this issue.

We are currently investigating reports of slow system performance impacting some users. Our team is working to identify the cause and restore normal performance as quickly as possible. We will provide further updates as they become available.