Kubernetes pods failing in AZ-EASTUS-1

CDF has recovered again after recurrence of the DNS issue.

Microsoft has confirmed they have now completed the rollback of the suspected root cause and we will continue monitoring the issue for a few more hours to verify CDF has fully recovered.

While our previous update indicated a good state of recovery, we have started to observe service interruptions again across the environment.

Customers may continue to experience slowness or intermittent failures in applications as we navigate the final stages of stabilization following the DNS outage. Our engineering team is closely monitoring these new developments and working to ensure a consistent recovery.

We will provide further updates as the situation stabilizes.

Cognite has observed all services have recovered or are in a good state of recovery. Customers may possibly still see some slowness in apps recovering from the dns outage.

We have confirmed that all customers on az-eastus-1 are affected due to ongoing Azure platform issues in the East US region, now officially acknowledged by Microsoft. These issues impact provisioning, scaling, and connectivity for workloads, resulting in widespread DNS resolution failures and a large number of pods across multiple namespaces in CrashLoopBackOff.


Customer impact includes service disruptions for timeseries and data modeling services, intermittent errors accessing InField, and degraded availability for multiple workloads and endpoints. Example error patterns include Postgres connection timeouts and DNS lookup failures.

We are seeing widespread DNS resolution failures in the az-eastus-1 cluster, affecting multiple pods and disrupting the timeseries service.

Cognite is investigating k8s pod restarts in az-eastus-1. This is possibly affecting time series, searching, and contextualization capabilities. Updates will be provided when more information is gathered.