[NL-AMS1]: Storage backends issue

This incident has been resolved.

We thank you for your continued patience throughout this process and apologise for the inconvenience this has caused.

We are currently compiling a detailed post-mortem to outline the sequence of events, the steps taken to rectify the situation, and the measures we are implementing to prevent such occurrences in the future. This post-mortem report will be made available to the customers in the coming days.

We understand the gravity of this disruption and extend our sincerest apologies to all affected customers.

Furthermore, we acknowledge the fact that we didn't meet our customers expectations on the reliability and vow to do better.

Our restoration efforts to bring affected services online are now in the final stages, with nearly all remaining customer services operational. Our team is conducting thorough checks to ensure stability across all systems to complete the recovery of services.

We thank you for your continued support and understanding during this period.

We are committed to maintaining transparent communication and will provide the next comprehensive update at 22:00 UTC or as soon as the services are fully available - whichever is sooner.

We are making significant headway and remain committed to resolving the remaining issues as swiftly as possible. Our technical team is diligently working to bring the remaining systems back online.

We appreciate your patience and understanding as we push towards fully recovering services.

The next update will be provided at 20:15 UTC

Update: We are making steady progress in resolving the storage backend issue at our NL-AMS1 data centre. Our team is working tirelessly to restore all affected services.

We anticipate performing emergency maintenance to parts of the infrastructure, which may result in brief downtime for some customers. Affected customers will be notified individually.

We appreciate your patience and understanding as we push towards recovering all services. Our team remains all hands on deck, ensuring swift recovery of the affected customers services.

The next update will be shared at 18:15 UTC

We continue to make progress in resolving the storage backend issue at our NL-AMS1 datacentre. Our engineering team is fully engaged, with all hands on deck, diligently working to restore all services. While we have successfully reinstated several affected services, some VMs remain offline.

We estimate that it may take a few more hours to bring the remaining servers back online.
We appreciate your continued patience and understanding during this recovery phase.

If you require immediate assistance or have further inquiries, please do not hesitate to reach out to our 24/7 live support via the UpCloud Hub.

Thank you for your continued patience and trust in us as we work to resolve this matter.

Update: We are still working on the incident and we have brought back some of the affected services, we are still working on getting the rest of the affected VMs back online as soon as possible.

Update: we understand the cause of the problem and we have isolated the issue. We are working bring the affected services back online as soon as possible.

We have identified the cause of the problem and we are working to rectify it.

We are currently investigating a storage backend issue affecting multiple products within the NL-AMS1 datacentre.

This may impact Cloud Servers and EOL Object Storage. We recommend avoiding stopping/starting your server(s) at this time.

We are actively working to resolve this issue as swiftly as possible. We regret any inconvenience this might be causing and thank you for your patience as we work to address this.

If you have any questions or concerns, please feel free to contact our 24/74 live support in our control panel.