FR-PAR-2 connectivity loss

Switchover to main power has been completed successfully. We are back to a nominal state.

Datacenter teams are now confident the batteries were involved in the initial incident, they will attempt to switch back to main power at 14:30 CEST.

Batteries replacement has been completed on all 4 electrical paths.
Removed batteries are under tests and investigations to confirm their involvement in the initial incident.
No more action on production expected for now.
We are still on power generators for the time being.

Batteries replacement is ongoing, 1 of the 4 electrical paths has been done (no impact).
Datacenter teams will proceed with the next paths.

The datacenter provider has planned an intervention on Saturday at 10:30 CEST to replace batteries on the electrical paths that had issues today.
This operation will be done live and should have no impact.
Once replaced, old batteries will be inspected and tested to confirm they were the cause of the incident.
Power will be kept on generators during this operation and after until the root cause is fully confirmed

Updated status for impacted products:

- Object Storage: Everything back to normal
- Serveless containers and functions: Everything is back to normal.
- File: some instances may encounter troubles with their File Storage, we invite customers to restart their instances/nodes if affected

As the failover malfunction has not been diagnosed at this time, our datacenter manager will keep the affected elements under a generator.

Updated status for impacted products:

- Object Storage: still some latency issues and timeout on connecting some buckets
- Serveless containers and functions: Everything is back to normal. Situation recovered and stable since 02:40 PM
- File Storage: some instances may encounter troubles with their File Storage, we invite customers to restart their instances/nodes if affected

Updated status for impacted products:

- Kapsule: everything back to normal
- Databases: everything back to normal
- Object Storage: still some latency issues and timeout on connecting some buckets
- Serveless containers and functions: might face connection timeout issues when contacting their applications
- File Storage: some instances may encounter troubles with their File Storage, we invite customers to restart their instances/nodes if affected

Clarification: only network rooms are running on power generators, with enough fuel for a few days

Updated status for impacted products:

- Instances: everything back to normal
- Kapsule: same status
- Databases: same status
- Object Storage: same status
- Serveless containers and functions: might face connection timeout issues when contacting their applications

As the power failure root cause is not yet understood (datacenter is investigating with its providers), we encourage you to shift workload on alternatives regions/AZ if possible.

Datacenter informs us that we are running on power generators for now until the root cause is identified

Teams are working on the recovery. Here is the status of impacted products:

- Instances: you may encounter issues with l_ssd/block snapshots or volumes

- Kapsule: Control planes are reachable and in nominal state, you may need to verify that your controllers reconnected properly on the apiservers. You may have experienced node replacement due to the autoheal or autoscaling process, this may delay volume re-attachments.

- Databases: snapshots may be blocked, some failovers have been started for HA

- Object storage: you may encounter some latencies

Datacenter hosting FR-PAR-2 availability zone occured a complete power failure in operator rooms during a planned datacenter maintenance.
Both operator rooms (network connectivity) were powerless for a few minutes leading to a complete isolation of the availability zone during the issue. We are investigating with the datacenter provider to understand the root cause.”

Most services recovered and our teams are mobilized to recover the rest on fr-par-2. Root cause is a power issue in datacenter.
Other service on par1 & par3 were impacted during the service transition.

The issue is impacting multiple products on FR-PAR.

We are continuing to investigate this issue and searching for the root cause.

We are currently experiencing connection and network issues in the FR‑PAR region, affecting all FR‑PAR availability zones on Scaleway as well as DC5 on Dedibox.