29

In have scaled my Pods to 20 in my cluster and when I see the next day the few of the scaled Pods were recreated.

When I say Pod recreated, it is getting deleted and created freshly and the timestamp of the recreated Pod and the scaled Pods vary.

I was unable to find the reasons for the recreate of the Pods.

I could not find which Pod went for a recreate as the Pod is deleted and gone away. There are no logs in the journalctl regarding which Pod got recreated. Is there any way I can debug further to find the reason for the Pod recreate? Or what might be the reason for the PODs getting deleted?

Note: I have readiness and liveness probes defined, but these probes would act on container and would not lead to Pod recreate in my understanding.

Daniel Serodio
  • 293
Nikhil
  • 351

3 Answers3

18

Basically you need to check Pod's events (here is a tip how to do it).

Keep in mind, that events retention period is too short (approximately 1h), and you may need to store them somewhere else. Here is a good article how to do it with EFK stack

The most common reason of Pods recreation - Node unavailability. If Pod's Node is not available, Kubernetes recreates the Pods in other nodes.

Daniel Serodio
  • 293
Denis Romaniuk
  • 306
1

I am faicng the similiar issue with the pod recreate, with my situation experience, you can try this:

  • check the host kernel log and make sure there is no OOM killer events(using command dmesg -T | egrep -i 'killed process')
  • make sure your container proccess with pid 1 not exit, when the pid 1 ended the pod will exit, and kubelet will auto start the pod

I am facing the pod restart with this 2 situation.

Dolphin
  • 447
0

If a Node that any pod is scheduled to dies (drains out), those Pods are scheduled for deletion after a timeout period.

You can identify the reason for it by if the node that the pod was scheduled to was in a Ready state. Else the pod will go into a NotReady state and terminate after a period

Sakthi Sri S V
  • 11
  • 2