[fr-par-1] Latency Block Storage Issues

A BSSD volume cluster was put under heavy stress since 29/01/2025 - 12h30 UTC because of unusual number of volumes to purge. Issue has been diagnosed and patch applied at 29/01/2025 - 16h35 UTC to limit latencies impact.

Preventive measures have been taken to prevent another mishandling during this kind of maintenance.

Impact: high latencies on block storage in FR-PAR-1 since 29/01/2025 - 12h30 UTC until 29/01/2025 - 22h00 UTC.

99th latencies are back to acceptable levels. 99.95th may be still higher than usual for the next 10 hours until all queued operations are done.

We continue to monitor this issue.

Some process and servers are still stuck in high load. Latencies have been limited but are still high.
We are still working on it.

A BSSD volume cluster was put under heavy stress since 12h30 UTC because of unusual number of volumes to purge. It provoked high latencies on block storage in FR-PAR-1

Issue has been diagnosed and patch applied at 16h35 UTC to limit latencies impact.

Low latencies volumes (class SBS) are not impacted.

It impacts block storage users on fr-par-1. They can experiment high latencies. RDB is as well impacted by this issue.