Issue with RIPEstat

Our intervention worked and performance recovered.This incident has been resolved.

The previous configuration changed had a positive effect. Performance has recovered (90th percentile 80-120ms, error rate < 0.1%). We are continuing to monitor for any further issues.

We have made a number of configuration changes that appear to have improved the situation. We are monitoring the results.

RIPEstat is now consistently using the intended backend systems. However, the service is now affected by increased latency between our on-premise infrastructure and the rented bare-metal environment (~11ms increase). This is causing both higher response times and capacity pressures.

Current impact:

* The error rate is around 7-8%, which is significantly above our normal thresholds.
* Latency remains severely elevated.
* Median latency has increased from around 12 ms to around 75 ms.
* The 90th percentile has increased from around 200 ms to around 17 seconds.

We will carry out further interventions today to reduce latency, increase available capacity, and improve service stability. We will keep you updated.

The immediate fix has been applied and the service is now operating in a degraded state. Some queries and API endpoints remain unavailable or unreliable, and the service may continue to experience periods of overload.

We are continuing to monitor the situation and will work on further mitigations to improve stability and latency.

Old backend infrastructure was decommissioned on 4/Jun as part of the Backend Infrastructure Switch announced on May 27. A configuration was still referencing such infrastructure; the configuration has been updated and the service is now restored. We will monitor how the new backend service performs.

We are experiencing issues with RIPEstat. We are investigating and will update this announcement once we know more.