Pod Log Loss in k8s Cluster

Question

Problem Description

When the cluster creates a Pod, the logs are initially normal, and kubectl logs pod-name -n namespace can display normal logs;
After approximately 30 seconds to 1 minute (the time is uncertain), the following error can be seen in kubectl logs pod-name -n namespace:

failed to try resolving symlinks in path "/var/log/pods/default_dev-6c5c5c9fdb-rvghw_c9440f95-7d2c-4a60-bc5a-bb855fb40794/nginx-arm/0.log": lstat /var/log/pods/default_dev-6c5c5c9fdb-rvghw_c9440f95-7d2c-4a60-bc5a-bb855fb40794: no such file or directory

After approximately a few seconds, the kubectl logs pod-name -n namespace command returns empty, and no information can be seen.

log info

kubelet

Jan 26 15:29:28 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:29:28.208733   19797 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 114.114.114.114 119.29.29.29 240c::6666"
Jan 26 15:29:28 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:29:28.208955   19797 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 114.114.114.114 119.29.29.29 240c::6666"
Jan 26 15:30:16 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:30:16.208853   19797 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 114.114.114.114 119.29.29.29 240c::6666"
Jan 26 15:30:24 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:30:24.708535   19797 cri_stats_provider.go:690] "Unable to fetch container log stats" err="open /var/log/pods/kube-system_cilium-k9zf2_eac9cfdb-acc6-4f2b-b9b4-0a87ae4147cc/cilium-agent: no such file or directory" containerName="cilium-agent"
Jan 26 15:30:24 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:30:24.708622   19797 cri_stats_provider.go:690] "Unable to fetch container log stats" err="open /var/log/pods/kube-system_cilium-operator-f99b55df8-fcq6k_7c42bed4-d16b-4ac8-8bf1-b3edc9b4d61a/cilium-operator: no such file or directory" containerName="cilium-operator"
Jan 26 15:30:24 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:30:24.708653   19797 cri_stats_provider.go:690] "Unable to fetch container log stats" err="open /var/log/pods/kube-system_cilium-envoy-fj4b9_b1e3b256-cae7-4443-b07f-0133091d0bd8/cilium-envoy: no such file or directory" containerName="cilium-envoy"
Jan 26 15:30:29 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:30:29.208754   19797 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 114.114.114.114 119.29.29.29 240c::6666"
Jan 26 15:30:40 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:30:40.209106   19797 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 114.114.114.114 119.29.29.29 240c::6666"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912524   19797 kubelet.go:2474] "SyncLoop ADD" source="api" pods=["default/dev-6c5c5c9fdb-hf7kw"]
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912615   19797 cpu_manager.go:398] "RemoveStaleState: containerMap: removing container" podUID="dbbdcc05-a3d5-45d6-8b7a-9b8fa0106906" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912636   19797 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbbdcc05-a3d5-45d6-8b7a-9b8fa0106906" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912647   19797 cpu_manager.go:398] "RemoveStaleState: containerMap: removing container" podUID="e7360818-a115-4c4e-836f-a3afbd5f691f" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912655   19797 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7360818-a115-4c4e-836f-a3afbd5f691f" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912668   19797 cpu_manager.go:398] "RemoveStaleState: containerMap: removing container" podUID="b3124064-a728-43d7-b217-8a8087c81eb4" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912675   19797 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3124064-a728-43d7-b217-8a8087c81eb4" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912683   19797 cpu_manager.go:398] "RemoveStaleState: containerMap: removing container" podUID="35a112e1-aac8-49ba-bd63-e844bcc90f6c" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912690   19797 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a112e1-aac8-49ba-bd63-e844bcc90f6c" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912720   19797 memory_manager.go:355] "RemoveStaleState removing state" podUID="e7360818-a115-4c4e-836f-a3afbd5f691f" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912728   19797 memory_manager.go:355] "RemoveStaleState removing state" podUID="b3124064-a728-43d7-b217-8a8087c81eb4" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912735   19797 memory_manager.go:355] "RemoveStaleState removing state" podUID="dbbdcc05-a3d5-45d6-8b7a-9b8fa0106906" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.912741   19797 memory_manager.go:355] "RemoveStaleState removing state" podUID="35a112e1-aac8-49ba-bd63-e844bcc90f6c" containerName="nginx-arm"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.913015   19797 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="default/dev-6c5c5c9fdb-hf7kw"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.914994   19797 reflector.go:376] Caches populated for *v1.ConfigMap from object-"default"/"kube-root-ca.crt"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.917310   19797 kubelet.go:2474] "SyncLoop ADD" source="api" pods=["default/dev-6c5c5c9fdb-jm4d4"]
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.917570   19797 kubelet.go:2474] "SyncLoop ADD" source="api" pods=["default/dev-6c5c5c9fdb-wqmzr"]
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.917627   19797 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="default/dev-6c5c5c9fdb-jm4d4"
Jan 26 15:30:41 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:41.917867   19797 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="default/dev-6c5c5c9fdb-wqmzr"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.041009   19797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrtt\" (UniqueName: \"kubernetes.io/projected/563227ff-cae2-4873-8742-2a077d5c3a28-kube-api-access-bzrtt\") pod \"dev-6c5c5c9fdb-jm4d4\" (UID: \"563227ff-cae2-4873-8742-2a077d5c3a28\") " pod="default/dev-6c5c5c9fdb-jm4d4"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.041059   19797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2dz\" (UniqueName: \"kubernetes.io/projected/b04cae6c-ab39-4cf9-8cf5-d6e1326a6581-kube-api-access-pp2dz\") pod \"dev-6c5c5c9fdb-wqmzr\" (UID: \"b04cae6c-ab39-4cf9-8cf5-d6e1326a6581\") " pod="default/dev-6c5c5c9fdb-wqmzr"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.041129   19797 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgd6w\" (UniqueName: \"kubernetes.io/projected/f3d67b4b-5fe9-419f-b796-659b749ebd46-kube-api-access-pgd6w\") pod \"dev-6c5c5c9fdb-hf7kw\" (UID: \"f3d67b4b-5fe9-419f-b796-659b749ebd46\") " pod="default/dev-6c5c5c9fdb-hf7kw"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.141392   19797 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pgd6w\" (UniqueName: \"kubernetes.io/projected/f3d67b4b-5fe9-419f-b796-659b749ebd46-kube-api-access-pgd6w\") pod \"dev-6c5c5c9fdb-hf7kw\" (UID: \"f3d67b4b-5fe9-419f-b796-659b749ebd46\") " pod="default/dev-6c5c5c9fdb-hf7kw"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.141427   19797 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrtt\" (UniqueName: \"kubernetes.io/projected/563227ff-cae2-4873-8742-2a077d5c3a28-kube-api-access-bzrtt\") pod \"dev-6c5c5c9fdb-jm4d4\" (UID: \"563227ff-cae2-4873-8742-2a077d5c3a28\") " pod="default/dev-6c5c5c9fdb-jm4d4"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.141450   19797 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2dz\" (UniqueName: \"kubernetes.io/projected/b04cae6c-ab39-4cf9-8cf5-d6e1326a6581-kube-api-access-pp2dz\") pod \"dev-6c5c5c9fdb-wqmzr\" (UID: \"b04cae6c-ab39-4cf9-8cf5-d6e1326a6581\") " pod="default/dev-6c5c5c9fdb-wqmzr"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.150875   19797 operation_generator.go:614] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrtt\" (UniqueName: \"kubernetes.io/projected/563227ff-cae2-4873-8742-2a077d5c3a28-kube-api-access-bzrtt\") pod \"dev-6c5c5c9fdb-jm4d4\" (UID: \"563227ff-cae2-4873-8742-2a077d5c3a28\") " pod="default/dev-6c5c5c9fdb-jm4d4"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.151038   19797 operation_generator.go:614] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2dz\" (UniqueName: \"kubernetes.io/projected/b04cae6c-ab39-4cf9-8cf5-d6e1326a6581-kube-api-access-pp2dz\") pod \"dev-6c5c5c9fdb-wqmzr\" (UID: \"b04cae6c-ab39-4cf9-8cf5-d6e1326a6581\") " pod="default/dev-6c5c5c9fdb-wqmzr"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.152241   19797 operation_generator.go:614] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgd6w\" (UniqueName: \"kubernetes.io/projected/f3d67b4b-5fe9-419f-b796-659b749ebd46-kube-api-access-pgd6w\") pod \"dev-6c5c5c9fdb-hf7kw\" (UID: \"f3d67b4b-5fe9-419f-b796-659b749ebd46\") " pod="default/dev-6c5c5c9fdb-hf7kw"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.244456   19797 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="default/dev-6c5c5c9fdb-hf7kw"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.249126   19797 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="default/dev-6c5c5c9fdb-wqmzr"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.251865   19797 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="default/dev-6c5c5c9fdb-jm4d4"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.873135   19797 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="default/dev-6c5c5c9fdb-jm4d4" event={"ID":"563227ff-cae2-4873-8742-2a077d5c3a28","Type":"ContainerStarted","Data":"bcc32955d3a1b14fcdb9d382d24f635f5860564018435ac2f806cecccb92c278"}
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.874555   19797 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="default/dev-6c5c5c9fdb-wqmzr" event={"ID":"b04cae6c-ab39-4cf9-8cf5-d6e1326a6581","Type":"ContainerStarted","Data":"7b6691c44adecb0caf2d2d01b4a6a7e5d06b7acbe756a3a22e4db91a8e787eb5"}
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.874600   19797 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="default/dev-6c5c5c9fdb-wqmzr" event={"ID":"b04cae6c-ab39-4cf9-8cf5-d6e1326a6581","Type":"ContainerStarted","Data":"781def1a9a09ee40a9733cd8866756bd2f4f7f50151940bedc70a7c62ffa95fa"}
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.875825   19797 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="default/dev-6c5c5c9fdb-hf7kw" event={"ID":"f3d67b4b-5fe9-419f-b796-659b749ebd46","Type":"ContainerStarted","Data":"f5116685550ae8dc57d2d68f15362ba5573c7718163fe7bb9a1ef33a74abe431"}
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.890769   19797 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="default/dev-6c5c5c9fdb-wqmzr" podStartSLOduration=1.890753568 podStartE2EDuration="1.890753568s" podCreationTimestamp="2025-01-26 15:30:41 +0800 CST" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-01-26 15:30:42.882841016 +0800 CST m=+248479.817543967" watchObservedRunningTime="2025-01-26 15:30:42.890753568 +0800 CST m=+248479.825456509"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:42.891241   19797 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="default/dev-6c5c5c9fdb-hf7kw" podStartSLOduration=1.891223453 podStartE2EDuration="1.891223453s" podCreationTimestamp="2025-01-26 15:30:41 +0800 CST" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-01-26 15:30:42.890626352 +0800 CST m=+248479.825329303" watchObservedRunningTime="2025-01-26 15:30:42.891223453 +0800 CST m=+248479.825926404"
Jan 26 15:30:43 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:43.879136   19797 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="default/dev-6c5c5c9fdb-jm4d4" event={"ID":"563227ff-cae2-4873-8742-2a077d5c3a28","Type":"ContainerStarted","Data":"11698c02c99d7b254462ae33f1a694ccd3209b99dbae0ea501324c627e901ed0"}
Jan 26 15:30:43 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:43.880470   19797 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="default/dev-6c5c5c9fdb-hf7kw" event={"ID":"f3d67b4b-5fe9-419f-b796-659b749ebd46","Type":"ContainerStarted","Data":"db52abe9586c40a9360c8470d26f5b8511b75e2c119be9f35126c615251debbc"}
Jan 26 15:30:43 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:30:43.889187   19797 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="default/dev-6c5c5c9fdb-jm4d4" podStartSLOduration=2.889175527 podStartE2EDuration="2.889175527s" podCreationTimestamp="2025-01-26 15:30:41 +0800 CST" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-01-26 15:30:43.88904933 +0800 CST m=+248480.823752291" watchObservedRunningTime="2025-01-26 15:30:43.889175527 +0800 CST m=+248480.823878468"
Jan 26 15:31:09 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:31:09.366679   19797 log.go:25] "Finished parsing log file" path="/var/log/pods/default_dev-6c5c5c9fdb-wqmzr_b04cae6c-ab39-4cf9-8cf5-d6e1326a6581/nginx-arm/0.log"
Jan 26 15:31:15 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:31:15.217668   19797 log.go:25] "Finished parsing log file" path="/var/log/pods/default_dev-6c5c5c9fdb-wqmzr_b04cae6c-ab39-4cf9-8cf5-d6e1326a6581/nginx-arm/0.log"
Jan 26 15:31:17 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:31:17.297096   19797 log.go:25] "Finished parsing log file" path="/var/log/pods/default_dev-6c5c5c9fdb-wqmzr_b04cae6c-ab39-4cf9-8cf5-d6e1326a6581/nginx-arm/0.log"
Jan 26 15:31:18 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:31:18.209034   19797 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 114.114.114.114 119.29.29.29 240c::6666"
Jan 26 15:31:19 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:31:19.374816   19797 log.go:25] "Finished parsing log file" path="/var/log/pods/default_dev-6c5c5c9fdb-wqmzr_b04cae6c-ab39-4cf9-8cf5-d6e1326a6581/nginx-arm/0.log"
Jan 26 15:31:21 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:31:21.449710   19797 log.go:25] "Finished parsing log file" path="/var/log/pods/default_dev-6c5c5c9fdb-wqmzr_b04cae6c-ab39-4cf9-8cf5-d6e1326a6581/nginx-arm/0.log"
Jan 26 15:31:25 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:31:25.132571   19797 cri_stats_provider.go:690] "Unable to fetch container log stats" err="open /var/log/pods/default_dev-6c5c5c9fdb-hf7kw_f3d67b4b-5fe9-419f-b796-659b749ebd46/nginx-arm: no such file or directory" containerName="nginx-arm"
Jan 26 15:31:25 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:31:25.132626   19797 cri_stats_provider.go:690] "Unable to fetch container log stats" err="open /var/log/pods/kube-system_cilium-k9zf2_eac9cfdb-acc6-4f2b-b9b4-0a87ae4147cc/cilium-agent: no such file or directory" containerName="cilium-agent"
Jan 26 15:31:25 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:31:25.132651   19797 cri_stats_provider.go:690] "Unable to fetch container log stats" err="open /var/log/pods/kube-system_cilium-operator-f99b55df8-fcq6k_7c42bed4-d16b-4ac8-8bf1-b3edc9b4d61a/cilium-operator: no such file or directory" containerName="cilium-operator"
Jan 26 15:31:25 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:31:25.132675   19797 cri_stats_provider.go:690] "Unable to fetch container log stats" err="open /var/log/pods/kube-system_cilium-envoy-fj4b9_b1e3b256-cae7-4443-b07f-0133091d0bd8/cilium-envoy: no such file or directory" containerName="cilium-envoy"
Jan 26 15:31:25 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:31:25.132701   19797 cri_stats_provider.go:690] "Unable to fetch container log stats" err="open /var/log/pods/default_dev-6c5c5c9fdb-wqmzr_b04cae6c-ab39-4cf9-8cf5-d6e1326a6581/nginx-arm: no such file or directory" containerName="nginx-arm"
Jan 26 15:31:25 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: E0126 15:31:25.132718   19797 cri_stats_provider.go:690] "Unable to fetch container log stats" err="open /var/log/pods/default_dev-6c5c5c9fdb-jm4d4_563227ff-cae2-4873-8742-2a077d5c3a28/nginx-arm: no such file or directory" containerName="nginx-arm"
Jan 26 15:31:31 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn kubelet[19797]: I0126 15:31:31.866884   19797 log.go:25] "Finished parsing log file" path="/var/log/pods/default_dev-6c5c5c9fdb-wqmzr_b04cae6c-ab39-4cf9-8cf5-d6e1326a6581/nginx-arm/0.log"

containerd

Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.513522648+08:00" level=warning msg="\"io.containerd.runtime.v1.linux\" is deprecated since containerd v1.4 and will be removed in containerd v2.0, use \"io.containerd.runc.v2\" instead"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.514602628+08:00" level=info msg="shim containerd-shim started" address="unix:///run/containerd/s/df0ca7e83e98dfd9e1bc1d2ff7a9a4b561aa2def7818c68a3c0cbfabf986da00" debug=false error="<nil>" id=781def1a9a09ee40a9733cd8866756bd2f4f7f50151940bedc70a7c62ffa95fa namespace=k8s.io pid=45128
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42+08:00" level=info msg="unable to get netns cookie" args="IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=dev-6c5c5c9fdb-jm4d4;K8S_POD_INFRA_CONTAINER_ID=bcc32955d3a1b14fcdb9d382d24f635f5860564018435ac2f806cecccb92c278;K8S_POD_UID=563227ff-cae2-4873-8742-2a077d5c3a28" containerID=bcc32955d3a1b14fcdb9d382d24f635f5860564018435ac2f806cecccb92c278 error="protocol not available" eventID=55169e68-b738-4879-9544-4754fd6e19b9 file-path=/opt/cni/bin ifName=eth0 k8sNamespace=default k8sPodName=dev-6c5c5c9fdb-jm4d4 netns=/var/run/netns/cni-4641d76c-000d-dd09-11e2-9e49e2c04dfc subsys=cilium-cni
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.582196354+08:00" level=warning msg="\"io.containerd.runtime.v1.linux\" is deprecated since containerd v1.4 and will be removed in containerd v2.0, use \"io.containerd.runc.v2\" instead"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.583170556+08:00" level=info msg="shim containerd-shim started" address="unix:///run/containerd/s/a86efe5397b2ffe5c16dc21f4325b4784c4b975f5f9ae3603e4834197ddadca7" debug=false error="<nil>" id=bcc32955d3a1b14fcdb9d382d24f635f5860564018435ac2f806cecccb92c278 namespace=k8s.io pid=45175
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.682736029+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:dev-6c5c5c9fdb-hf7kw,Uid:f3d67b4b-5fe9-419f-b796-659b749ebd46,Namespace:default,Attempt:0,} returns sandbox id \"f5116685550ae8dc57d2d68f15362ba5573c7718163fe7bb9a1ef33a74abe431\""
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.685035968+08:00" level=info msg="CreateContainer within sandbox \"f5116685550ae8dc57d2d68f15362ba5573c7718163fe7bb9a1ef33a74abe431\" for container &ContainerMetadata{Name:nginx-arm,Attempt:0,}"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.689754558+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:dev-6c5c5c9fdb-wqmzr,Uid:b04cae6c-ab39-4cf9-8cf5-d6e1326a6581,Namespace:default,Attempt:0,} returns sandbox id \"781def1a9a09ee40a9733cd8866756bd2f4f7f50151940bedc70a7c62ffa95fa\""
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.691788250+08:00" level=info msg="CreateContainer within sandbox \"781def1a9a09ee40a9733cd8866756bd2f4f7f50151940bedc70a7c62ffa95fa\" for container &ContainerMetadata{Name:nginx-arm,Attempt:0,}"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.717638489+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:dev-6c5c5c9fdb-jm4d4,Uid:563227ff-cae2-4873-8742-2a077d5c3a28,Namespace:default,Attempt:0,} returns sandbox id \"bcc32955d3a1b14fcdb9d382d24f635f5860564018435ac2f806cecccb92c278\""
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.719702037+08:00" level=info msg="CreateContainer within sandbox \"bcc32955d3a1b14fcdb9d382d24f635f5860564018435ac2f806cecccb92c278\" for container &ContainerMetadata{Name:nginx-arm,Attempt:0,}"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.729281235+08:00" level=info msg="CreateContainer within sandbox \"781def1a9a09ee40a9733cd8866756bd2f4f7f50151940bedc70a7c62ffa95fa\" for &ContainerMetadata{Name:nginx-arm,Attempt:0,} returns container id \"7b6691c44adecb0caf2d2d01b4a6a7e5d06b7acbe756a3a22e4db91a8e787eb5\""
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.729845442+08:00" level=info msg="StartContainer for \"7b6691c44adecb0caf2d2d01b4a6a7e5d06b7acbe756a3a22e4db91a8e787eb5\""
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.730530030+08:00" level=warning msg="\"io.containerd.runtime.v1.linux\" is deprecated since containerd v1.4 and will be removed in containerd v2.0, use \"io.containerd.runc.v2\" instead"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.731505802+08:00" level=info msg="shim containerd-shim started" address="unix:///run/containerd/s/9c6a0621caefafe2f8d577d5c0c53c01eb2f5777fae96d7eee3b0cc95993a740" debug=false error="<nil>" id=7b6691c44adecb0caf2d2d01b4a6a7e5d06b7acbe756a3a22e4db91a8e787eb5 namespace=k8s.io pid=45215
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.740159803+08:00" level=info msg="CreateContainer within sandbox \"f5116685550ae8dc57d2d68f15362ba5573c7718163fe7bb9a1ef33a74abe431\" for &ContainerMetadata{Name:nginx-arm,Attempt:0,} returns container id \"db52abe9586c40a9360c8470d26f5b8511b75e2c119be9f35126c615251debbc\""
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.740821402+08:00" level=info msg="StartContainer for \"db52abe9586c40a9360c8470d26f5b8511b75e2c119be9f35126c615251debbc\""
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.741380142+08:00" level=warning msg="\"io.containerd.runtime.v1.linux\" is deprecated since containerd v1.4 and will be removed in containerd v2.0, use \"io.containerd.runc.v2\" instead"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.742398673+08:00" level=info msg="shim containerd-shim started" address="unix:///run/containerd/s/ee66c1327fc9b55d4750b9fc1fb7ff19cdbe97e56164519e9535a5adbe410cd6" debug=false error="<nil>" id=db52abe9586c40a9360c8470d26f5b8511b75e2c119be9f35126c615251debbc namespace=k8s.io pid=45231
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.763629552+08:00" level=info msg="CreateContainer within sandbox \"bcc32955d3a1b14fcdb9d382d24f635f5860564018435ac2f806cecccb92c278\" for &ContainerMetadata{Name:nginx-arm,Attempt:0,} returns container id \"11698c02c99d7b254462ae33f1a694ccd3209b99dbae0ea501324c627e901ed0\""
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.764417927+08:00" level=info msg="StartContainer for \"11698c02c99d7b254462ae33f1a694ccd3209b99dbae0ea501324c627e901ed0\""
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.765131550+08:00" level=warning msg="\"io.containerd.runtime.v1.linux\" is deprecated since containerd v1.4 and will be removed in containerd v2.0, use \"io.containerd.runc.v2\" instead"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.766304504+08:00" level=info msg="shim containerd-shim started" address="unix:///run/containerd/s/3fb65c91b14182d8342af4583de15422322a017075974811af32ac5a46afcca7" debug=false error="<nil>" id=11698c02c99d7b254462ae33f1a694ccd3209b99dbae0ea501324c627e901ed0 namespace=k8s.io pid=45253
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.859584410+08:00" level=info msg="StartContainer for \"7b6691c44adecb0caf2d2d01b4a6a7e5d06b7acbe756a3a22e4db91a8e787eb5\" returns successfully"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.872137581+08:00" level=info msg="StartContainer for \"db52abe9586c40a9360c8470d26f5b8511b75e2c119be9f35126c615251debbc\" returns successfully"
Jan 26 15:30:42 dev-nm-huhehaote5-devopstest-33.in.ctcdn.cn containerd[13020]: time="2025-01-26T15:30:42.889480525+08:00" level=info msg="StartContainer for \"11698c02c99d7b254462ae33f1a694ccd3209b99dbae0ea501324c627e901ed0\" returns successfully"

Environment Information

In the ARM physical machine of the test environment, we created a cluster using systemd to start the k8s component, which was obtained through the changelog of k8s
os-release CentOS Linux
kernel: 4.18.0-3.2.el7.aarch64
kubernetes version v1.32.1
ctr version v1.7.25

I tried to check kubelet's configuration, kernel parameters of the nodes, and compared with another set of x86 clusters; No problem found This is my first time to ask a question in this community. I'm sorry that my English expression may not be very good; I've looked up a lot of information and tried a lot Any help would be greatly appreciated

song dashan · Answer 1 · 2025-02-06T07:20:33.450

configration for kubelet and containerd

ps -ef | grep kubelet | grep -v grep

root     88942     1  2 Jan26 ?        05:28:48 /usr/local/bin/kubelet --config=/etc/kubernetes/kubelet-config.yaml --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --hostname-override=182.42.233.161 --pod-infra-container-image=harbor.ctyuncdn.cn/devops/pause-arm:3.10 --root-dir=/var/lib/kubelet --cert-dir=/etc/kubernetes/ssl --v=2

kubelet-config

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/ssl/kube-ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.1.0.10
clusterDomain: cluster.local
configMapAndSecretChangeDetectionStrategy: Watch
containerLogMaxFiles: 50
containerLogMaxSize: 10Mi
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 300Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 40s
hairpinMode: hairpin-veth
healthzBindAddress: 0.0.0.0
healthzPort: 10248
httpCheckFrequency: 40s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
kubeAPIBurst: 100
kubeAPIQPS: 50
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeLeaseDurationSeconds: 40
nodeStatusReportFrequency: 1m0s
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
port: 10250
# disable readOnlyPort
readOnlyPort: 0
resolvConf: /etc/resolv.conf
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
tlsCertFile: /etc/kubernetes/ssl/kubelet.pem
tlsPrivateKeyFile: /etc/kubernetes/ssl/kubelet-key.pem

cat /etc/containerd/config.toml

version = 2
[plugins."io.containerd.grpc.v1.cri"]
  enable_selinux = false
  sandbox_image = "harbor.ctyuncdn.cn/devops/pause-arm:3.10"
enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runtime.v1.linux"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

and log file info

[root@dev-nm-huhehaote5-devopstest-33 pods]# ls */*/ -alh
default_dev-6c5c5c9fdb-69kqs_84e1fc81-264b-4af6-a6dc-69a5800961e6/nginx-arm/:
total 0
drwxr-xr-x 2 root root 19 Feb  6 15:05 .
drwxr-xr-x 3 root root 23 Feb  6 15:05 ..
-rw-r----- 1 root root  0 Feb  6 15:05 0.log
default_dev-6c5c5c9fdb-9q7wj_3ec5eb33-3c4c-47c8-9581-c5aa7ade39f3/nginx-arm/:
total 0
drwxr-xr-x 2 root root 19 Feb  6 15:05 .
drwxr-xr-x 3 root root 23 Feb  6 15:05 ..
-rw-r----- 1 root root  0 Feb  6 15:05 0.log
default_dev-6c5c5c9fdb-f9jkp_9533fcd4-30a1-4f44-b092-26c2329d0b61/nginx-arm/:
total 0
drwxr-xr-x 2 root root 19 Feb  6 15:05 .
drwxr-xr-x 3 root root 23 Feb  6 15:05 ..
-rw-r----- 1 root root  0 Feb  6 15:05 0.log
default_dev-6c5c5c9fdb-rvghw_c9440f95-7d2c-4a60-bc5a-bb855fb40794/nginx-arm/:
total 0
drwxr-xr-x 2 root root 19 Feb  6 15:05 .
drwxr-xr-x 3 root root 23 Feb  6 15:05 ..
-rw-r----- 1 root root  0 Feb  6 15:05 0.log
kube-system_cilium-envoy-fj4b9_b1e3b256-cae7-4443-b07f-0133091d0bd8/cilium-envoy/:
total 0
drwxr-xr-x 2 root root 19 Feb  6 15:05 .
drwxr-xr-x 3 root root 26 Feb  6 15:05 ..
-rw-r----- 1 root root  0 Feb  6 15:05 0.log
kube-system_cilium-k9zf2_eac9cfdb-acc6-4f2b-b9b4-0a87ae4147cc/cilium-agent/:
total 0
drwxr-xr-x 2 root root 19 Feb  6 15:05 .
drwxr-xr-x 3 root root 26 Feb  6 15:05 ..
-rw-r----- 1 root root  0 Feb  6 15:05 0.log
kube-system_cilium-operator-f99b55df8-fcq6k_7c42bed4-d16b-4ac8-8bf1-b3edc9b4d61a/cilium-operator/:
total 0
drwxr-xr-x 2 root root 19 Feb  6 15:05 .
drwxr-xr-x 3 root root 29 Feb  6 15:05 ..
-rw-r----- 1 root root  0 Feb  6 15:05 0.log
[root@dev-nm-huhehaote5-devopstest-33 pods]# pwd
/var/log/pods