MK8s: Partial Connectivity Degradation to Control Planes

This incident is now marked resolved, as all affected control planes have returned to and stayed in a stable state.
A Root Cause Analysis (RCA) is underway and will be published here once finalized.

Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

The changes applied to some control plane clusters have had positive effects. The team is continuing the rollout to other affected clusters.

We are marking DBaaS as recovered.
Our Kubernetes Team is currently working on stabilizing the Kubernetes Control Plane. We are focusing on mitigating recurring load spikes influencing stability.

We are marking the Container Registry Service as recovered.

We are closing the incident for the AI Model Hub. All metrics have recovered and the service should be up and running again normally.

We are adding the Container Registry as an affected Service. Customers may currently experience issues pulling and pushing images from the Registry.

Our Kubernetes Team has deployed a fix for the affected AI Model Hub Database Services. We currently see metrics improving and monitoring the situation closely.

We are expanding the scope of this incident to include DBaaS and AI Model Hub. We have observed an increased error count originating from PostgresDB on Kubernetes. Additionally, to improve transparency, the previously reported separate incident regarding the AI Model Hub (https://status.ionos.cloud/incidents/rmgs845klm32) is being merged into this primary incident.

The team has identified the root cause as a resource constraint within the etcd database. Mitigation efforts are currently underway.

Some customers may experience connection problems to the control plane and degraded functionality of kubernetes.
Our teams are investigating and working on a resolution.