IONOS Cloud Outage History

IONOS Cloud's AI Model Hub experienced increased error rates with the Llama 405B model due to hardware degradation and subsequent capacity constraints following a hardware failure. The incident affected the Llama 3.1 405B Instruct model's performance and reliability over 58.9 hours. The service was restored with capacity constraints remaining, and users experiencing ongoing issues were advised to use GPT-OSS 120B as a temporary alternative while optimizations are deployed.

The IONOS Container Registry experienced increased errors for 7.3 hours, preventing customers from pulling container images due to an issue in the underlying Kubernetes cluster affecting a subset of images. The problem was resolved by rolling back a component of the Kubernetes control plane to a previous version. The service was restored and marked as resolved with no further issues detected.

IONOS Cloud experienced frontend issues in their Data Center Designer (DCD) that prevented users from editing Virtual Data Centers (VDCs), with affected customers encountering "In Progress/Pending" status messages. The problem was caused by a faulty DCD release that disrupted the user interface functionality. The issue was resolved after 2.5 hours by rolling back the problematic release, restoring normal VDC editing capabilities.

IONOS Cloud experienced a 10.2-hour network service degradation in their TXL data center that affected multiple services including Data Center Designer, Network, Provisioning, DBaaS, and AI Model Hub, with customers experiencing connectivity issues, extended provisioning times, and service outages. The incident was linked to ongoing maintenance work and required temporarily pausing provisioning operations to clear a backlog of queued jobs. All services were restored to full operational status with the provisioning backlog cleared, and a comprehensive root cause analysis was initiated.

IONOS Cloud experienced instability affecting Managed Kubernetes clusters in Frankfurt, where some clusters became stuck in updating/deploying states and suffered intermittent communication issues between control plane components and worker nodes. This caused brief service interruptions and degraded performance for impacted Kubernetes services. The incident was resolved after 2.9 hours of active investigation and remediation by engineering teams.

IONOS Cloud's Acronis Backup Service experienced connectivity issues for 12.5 hours, preventing users from accessing cloud storage with "access denied" errors. The problem was caused by a configuration change made on February 25th that affected backup connectivity. The issue was resolved by rolling back the configuration change, restoring normal backup service functionality.

IONOS Cloud experienced sporadic connection issues and substantial packet delivery delays that affected network connectivity and degraded connection quality for individual virtual resources. The network problems lasted 1.1 hours and were linked to scheduled network maintenance. The issue was resolved after technicians identified the source, implemented a fix, and restored normal traffic and connectivity.