Outage in Cloudera

Azure Kubernetes (AKS) Cluster Provisioning fails if the subnet(s) in use has an associated user-assigned NAT Gateway.

Resolved Minor
May 18, 2022 - Started almost 2 years ago - Lasted about 2 hours
Official incident page

Need to monitor Cloudera outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Cloudera, and never miss an outage again.
Start Free Trial

Outage Details

Azure Kubernetes (AKS) Cluster Provisioning for CDP Data Services such as CML(Cloudera Machine Learning, CDE(Cloudera Data Engineering), and CDF(Cloudera Data Flow) will fail if the subnet(s) in use has an associated NAT Gateway. Microsoft made a change starting from May, wherein creating an AKS cluster using the above Networking setup will fail unless the parameter: outboundType is set to userAssignedNATGateway. The error is: status: CREATE_FAILED message: '[error creating cluster] info: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="SubnetsAssociatedWithNATgatewayWhenOutboundTypeIsStandardLoadBalancer" Message="Subnets ''/subscriptions/017b8edd-1cb1-4c3e-aa2a-2f62880b1d3d/resourceGroups/liftie-test/providers/Microsoft.Network/virtualNetworks/liftie-dev/subnets/liftie-dev.internal.0.westus2, /subscriptions/017b8edd-1cb1-4c3e-aa2a-2f62880b1d3d/resourceGroups/liftie-test/providers/Microsoft.Network/virtualNetworks/liftie-dev/subnets/liftie-dev.internal.0.westus2'' should not have a NAT gateway associated when outbound type is SLB." Target="agentPoolProfile.vnetSubnetID"' Before the recent change, Azure accepted the default value for outboundType as loadBalancer and allowed the AKS cluster creation. Note that the above issue is not seen: * If the subnet(s) doesn't have a user-assigned NAT Gateway associated, - On existing clusters, and - On the clusters post-upgrade. References: - Creating AKS cluster with user-assigned NAT Gateway: https://docs.microsoft.com/en-us/azure/aks/nat-gateway#create-an-aks-cluster-with-a-user-assigned-nat-gateway - AKS Release Note: https://github.com/Azure/AKS/commit/dfd3e6176bc979d04ef5592513cd0d2ece5146df - Determining a possible workaround to be recommended is currently in progress. - Microsoft Case # 2205170010003095 was created as a follow-up.
Latest Updates ( sorted recent to last )
IDENTIFIED almost 2 years ago - at 05/18/2022 12:36AM

Azure Kubernetes (AKS) Cluster Provisioning for CDP Data Services such as CML(Cloudera Machine Learning, CDE(Cloudera Data Engineering), and CDF(Cloudera Data Flow) will fail if the subnet(s) in use has an associated NAT Gateway.

Microsoft made a change starting from May, wherein creating an AKS cluster using the above Networking setup will fail unless the parameter: outboundType is set to userAssignedNATGateway.

The error is:
status: CREATE_FAILED message: '[error creating cluster] info: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="SubnetsAssociatedWithNATgatewayWhenOutboundTypeIsStandardLoadBalancer" Message="Subnets ''/subscriptions/017b8edd-1cb1-4c3e-aa2a-2f62880b1d3d/resourceGroups/liftie-test/providers/Microsoft.Network/virtualNetworks/liftie-dev/subnets/liftie-dev.internal.0.westus2, /subscriptions/017b8edd-1cb1-4c3e-aa2a-2f62880b1d3d/resourceGroups/liftie-test/providers/Microsoft.Network/virtualNetworks/liftie-dev/subnets/liftie-dev.internal.0.westus2'' should not have a NAT gateway associated when outbound type is SLB." Target="agentPoolProfile.vnetSubnetID"'

Before the recent change, Azure accepted the default value for outboundType as loadBalancer and allowed the AKS cluster creation.

Note that the above issue is not seen: * If the subnet(s) doesn't have a user-assigned NAT Gateway associated,
- On existing clusters, and
- On the clusters post-upgrade.

References:
- Creating AKS cluster with user-assigned NAT Gateway: https://docs.microsoft.com/en-us/azure/aks/nat-gateway#create-an-aks-cluster-with-a-user-assigned-nat-gateway
- AKS Release Note: https://github.com/Azure/AKS/commit/dfd3e6176bc979d04ef5592513cd0d2ece5146df
- Determining a possible workaround to be recommended is currently in progress.
- Microsoft Case # 2205170010003095 was created as a follow-up.

The easiest way to monitor Cloudera and all cloud vendors

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3153 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime