What is the relationship between deployment and container level readiness probes?

Question

I have a helm chart that defines my container's readinessProbe as:

readinessProbe:
    httpGet:
      path: /health/readiness
      port: http

And when I deploy it I can browse to http://myservice.mydomain/istioBasePath/health/readiness and I get a response of Healthy.

But if I do a kubectl describe pod on one of the pods that comprise this service, I see this in the details for my container:

Readiness:  http-get http://:15020/app-health/my-servce-name-is-shown-here/readyz

Which is not at all what I setup in the deployment for my container. (If nothing else, it has a z in the name.)

Additionally, when I add the IP address of the container to that URL and run a curl from within the cluster, it does not respond with any text. Running curl with the -I option shows:

HTTP/1.1 200 OK
Date: Wed, 17 May 2023 23:26:11 GMT

(Again, no Healthy text is returned.)

Something is different here, but I am not quite sure what it is. The URL for my "Deployment Defined" readiness probe works, but it returns the text Healthy. The probe defined on the container (that I did not setup, and thought I was overriding with my deployment), just has a Http status result.

I thought I had only one probe going on, but clearly there is a difference.

Can someone explain the difference to me?

score 1 · Accepted Answer · answered Jun 24 '23 at 14:35

You are most likely using Istio which you did not mention in the question. By default (from version 1.5.2) probe rewrites are enabled. This is required because of mTLS - for more details read the documentation.

The health check requests to the liveness-http service are sent by Kubelet. This becomes a problem when mutual TLS is enabled, because the Kubelet does not have an Istio issued certificate. Therefore the health check requests will fail.

TCP probe checks need special handling, because Istio redirects all incoming traffic into the sidecar, and so all TCP ports appear open. The Kubelet simply checks if some process is listening on the specified port, and so the probe will always succeed as long as the sidecar is running.

Istio solves both these problems by rewriting the application PodSpec readiness/liveness probe, so that the probe request is sent to the sidecar agent. For HTTP and gRPC requests, the sidecar agent redirects the request to the application and strips the response body, only returning the response code. For TCP probes, the sidecar agent will then do the port check while avoiding the traffic redirection.

What is the relationship between deployment and container level readiness probes?

1 Answers1