Are you considering upgrading your Azure Kubernetes Service (AKS) cluster from a version prior to 1.24.3? You might want to read about my recent experience before you do. What was meant to be a routine upgrade turned into a frustrating two-hour debugging session. Although all pods and containers were running smoothly, we couldn't connect to the cluster from the outside internet.

At first, we thought it might be an IP address or DNS issue, but it turned out to be something more complicated. We ran troubleshooting on the AKS Load Balancer and it told us that firewall ports were being blocked. However, this was a red herring, and the root cause of the issue was entirely different.

The problem ended up being a breaking change that AKS introduced after Kubernetes v1.24.3 where they now require an annotation in your nginx ingress controller.

Turns out, a lot of people were bit by this issue and you can see people discussing it on this post here: https://serverfault.com/questions/1110280/kubernetes-v1-24-3-upgrade-broke-ingress-nginx.

That post does not make it obvious on how to apply the fix directly into your AKS cluster, so this is how you do it via the Azure Portal.

Open up your AKS in your azure portal and go to Services and Ingresses and then click on your nginx ingress controller.

Then click YAML so you can edit the YAML configuration directly. Once there, add the following line under the annotations section.

service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz

Save and apply this yaml file and you are good to go.

It's important to note that if you install nginx ingress following the Azure documentation outlined here: https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli#basic-configuration then you likely ran this command:

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --create-namespace \
  --namespace $NAMESPACE \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz

If you used that command to spin up ingress in your AKS cluster, then you do not need to add the annotation line as described above because it should already be there. But we made the mistake of assuming it was already there because we spun up several AKS clusters using that method. But the one cluster that failed the upgrade was created at a slightly earlier time then the others so that annotation was omitted during creation.

We hope this helps you out of a jam!

Subscribe to the Raytha newsletter