Network Service Degradation in TXL

The Provisioning and DBaaS services have been restored to full operational status. The backlog has been cleared and no further service degradation is expected.
Our Network Team is currently preparing a comprehensive Root Cause Analysis (RCA) of this incident, which we will publish on this page. We anticipate releasing the complete analysis by tomorrow.
Thank you for your patience.

We are setting the DCD frontend back to 'Operational' now that the dependency on the provisioning side has been resolved. Customers should now be able to use the DCD via the web frontend again.

We are setting the incident to 'Monitoring' status and will ensure the successful recovery of the still affected services. Customers might still experience a performance impact on provisioning-related activities, such as modifying infrastructure resources in the DCD.

We recommend to postpone non-mission critical/urgent changes until the incident is marked as "Resolved"

We are still waiting for the provisioning backlog to be processed. Customers might face extended provisioning times. DCD web-frontend availability and DBaaS services remain affected due to their dependency on provisioning assignments. We will continue to update the status page, though likely at a reduced cadence as we monitor backlog consumption.

In parallel, a Root Cause Analysis (RCA) has been initiated for the triggering incident. We will share the RCA here as soon as it becomes available.

We are setting AI Model Hub back to "operational"

We are currently working on re-establishing connectivity for the supporting systems for the DCD to restore the service, as well.

We see that AI Model Hub service is recovering. We are closely monitoring request processing and reducing the severity of the impact for this service.

Network connectivity is improving. We will re-enable provisioning and monitor job execution as well as the rest of the affected services.

Our network team is still diagnosing ongoing connectivity issues that affect the referenced services.

We are upgrading the impact for the AI Model Hub.

We are temporarily pausing provisioning in a cluster currently experiencing a backlog of queued jobs. While running services remain unaffected, updates to existing entities will not be processed until provisioning is reactivated.

We have added DBaaS in the list of affected services due to increased error count. The team is aware and actively working on the service.

We are seeing issues again on DCD frontend, as well as AI Model Hub. Customers may still see connectivity issues with these services. Our Services Team are investigating.

We are seeing services recovering. We will monitor the progress over the next few minutes and update the status of the affected services.

We have added DCD and AI Modelhub to the list of affected services. Customers might experience connectivity issues for these services currently.

We have identified an issue related to the ongoing maintenance: https://status.ionos.cloud/incidents/zg4mpk9x724t
Our network team is currently working on restoring service to the affected components. Customers might experience intermittent network service degradation or outages.
We are upgrading the severity of the incident and will keep you informed on the progress of the recovery.

We are currently investigating monitoring alerts in TXL. We will keep you updated on our investigation.