[Object Storage] - [fr-par][nl-ams][pl-waw] High Latencies

This incident has been resolved.

The Elastic Metal product is now back to full function.
We are continuing to monitor for any further issues.

The situation is now stabilized and we put up mitigations to keep it in nominal state. The definitive fix is being developed and tested and will be deployed in the beginning of next week.
There should be no more impact but we will keep this status open to keep you updated.

Update:
The situation is now stabilized and we put up mitigations to keep it in nominal state. The definitive fix is being developed and tested and will be deployed in the beginning of next week.
There should be no more impact but we will keep this status open to keep you updated.

We are continuing to monitor for any further issues.

Update:
5xx error rate is back to nominal, but latencies are still high. Sadly the hotfix release did not fix every issue and our team is working on a patch that will be released tomorrow first thing in the morning.
There is still some impacts on registry product.

Update: The hotfix release is still in progress on all the region and will take a while. There was a slight raise of error rates (0.11%) arround 15:05 CEST due to a cluster member having some issues and necessitating a reboot and resync but quickly fixed.

Impacts on registry product are still monitored but show improvements.

The situation is improving.

AMS and WAW region's 500s error rate is less than 0.001% and stable.

For PAR region, the impact on instances and RDB products were reduced but must be improved to return to nominal state.
For registry product, object-storage backend latencies still implies some 500s due to timeouts.

The object-storage product team have prepared a new hotfix that is currently deployed on all PAR region. That action is estimated to take 7 hours to deploy at minimum.

But object-storage response times should progressively decrease during this timeframe on this region.
We are still closely monitoring this issue.

Update:
500s errors on object-storage rate dropped on all regions. PAR region still experiences high latencies.
Thoses latencies impacts Registry product by creating 500s due to timeouts.
We are still working in improving response time.

Update:
Impacts on ams and waw regions (500s errors) will settle very soon.

Update:
Latencies worsened this monday morning.
Service started show degradation degraded on ams region too.
All hands are on deck to fix this issue as soon as possible.

Since 00h00 UTC P95 latencies are rising again. We are working on mitigating the load.

The vast majority of Service latencies (P95) are back to acceptable levels thanks to the actions done by the object-storage team.
We are still working to further improve the P99s.

Identified : Thoses latencies are caused by the load on our AZ being unbalanced. That implies higher latencies for requests being routed to some servers of this AZ.
This issue also may cause latencies on others products (Registry and Serverless)

Our team is currently working on rebalancing the load. Situation have slightly improved since the start of this operation (14H UTC).
A more aggressive action is planned and will be applied soon to alleviate the load on the concerned servers.

Next update will be done in 2 hour or less.

The Object Storage service in the fr-par region is experiencing high latencies. This may result in slower access to stored objects. We are working to resolve the issue.