Scaling activities

The situation is stable. We will resolve this incident here and follow up with a post incident review in the coming days.

The original database cluster can only be started in read mode.

In accordance with our backup and recovery processes, we promoted the new database cluster with the state of 2024-01-11 03:05 UTC as the new production cluster. We updated all workloads to use this new database cluster. Please note that this does not contain data between 2024-01-11 03:05 UTC and the moment the database cluster went offline (~ 2024-01-11 07:22 UTC).

Dumps of the original database with the latest data can be exported and shared on request.

A summary of the incident will be shared in the upcoming days.

We are sorry for the inconvenience this caused you and your clients. If you have any questions regarding this, please reach out to us.

Recovering the database was interrupted due to an unforeseen issue. We are working with the AWS RDS team to bring the database back online.

As an alternative option for recovery we can point single environments to a new database cluster, containing data up until 2024-01-11 03:05 UTC. Please be aware that this option would lead to data loss. If you would like to pursue this route, please contact us through our support channels.

We're making good progress on recovering the database cluster. We're expecting the database cluster to be back online within the next 2 hours.

Recovery is still underway. We're evaluating additional ways to recover from the current situation quicker and restore services.

We're making progress in recovery - We can't give a firm ETA as the recovery speed hasn't settled fully yet. Still in discussions with the AWS RDS team on timings and additional recovery options.

We've identified the issue in the meantime and working on recovering from the outage. We can't give an ETA for now and evaluating several options.

We're still working with AWS RDS team to investigate the issue and what causes the connectivity issues.

We're seeing connectivity issues to the database cluster after the scaling operation.

We'll involve our upstream provider to look into this issue aswell.

We're seeing issues with the Database Cluster and investigating

We observed an increase in resource usage on the shared MySQL cluster on UK3. To account for the increase we are scaling the cluster which will lead to one failover.