14

After upgrading our AKS kubernetes cluster to from v1.23.8 to v1.24.3 our ingress stopped working properly. No errors logged in events and the ingress-nginx pod does not report any errors on the console. Everything looks fine from within the cluster, but all ports for the public IP is closed externally.

Even curl'ing the web-apps that run in the cluster from within the cluster works fine. It seems like it's just the opening of the ports externally that's broken. Ingress-nginx is deployed via helm release (HR v4.2.5).

I have a feeling it must be some config for the ingress or controller that needs to be changed.

UPDATE: we did a new install of a plain AKS cluster and did helm install quickstart ingress-nginx/ingress-nginx in 1.23.8 (which works), 1.24.0 (which does not work) and in 1.24.3 (which does not work either).

Any ideas or pointers?

sevenam
  • 441

2 Answers2

20

We found the issue.

For clusters v1.24.0 and up the health probes for the load balancer is set to HTTP and HTTPS instead of TCP. When we changed the health probes to use TCP it all worked again.

Created an issue for AKS on this: https://github.com/Azure/AKS/issues/3210

The proper fix was to add the following annotation to the nginx service (see link to AKS issue above):

values:
controller:
  service: 
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
sevenam
  • 441
0

Just want to highlight this note from Microsoft:

<<The Kubernetes API health endpoint, healthz was deprecated in Kubernetes v1.16. You can replace this endpoint with the livez and readyz endpoints instead.>>

Reference: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/load-bal-ingress-c/create-unmanaged-ingress-controller?tabs=azure-cli#before-you-begin

Note sure how it fixed your issue with a deprecated API

Said Amezyane
  • 1