kubernetes fails to pull image

Question

We are using kubernetes 1.30 when I deploy a pod that using an image from private repository it gives the following error:

Failed to pull image "dsaber.repo.farbeen.local/sad-rpc:0.0.1": failed to pull and unpack image "dsaber.repo.farbeen.local/sad-rpc:0.0.1": failed to resolve reference "dsaber.repo.farbeen.local/sad-rpc:0.0.1": unexpected status from HEAD request to http://dsaber.repo.farbeen.local/v2/sad-rpc/manifests/0.0.1: 502 Server dropped connection

I changed the configuration of containerd like this:

~# cat /etc/containerd/config.toml 
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0
[grpc]
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
[debug]
  level = "info"
[metrics]
  address = ""
  grpc_histogram = false
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "registry.k8s.io/pause:3.9"
    max_container_log_line_size = -1
    enable_unprivileged_ports = false
    enable_unprivileged_icmp = false
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      snapshotter = "overlayfs"
      discard_unpacked_layers = true
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          runtime_engine = ""
          runtime_root = ""
          base_runtime_spec = "/etc/containerd/cri-base.json"
      [plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd.runtimes.runc.options]
        systemdCgroup = true
        binaryName = &quot;/usr/local/bin/runc&quot;
[plugins.&quot;io.containerd.grpc.v1.cri&quot;.registry]
  config_path = &quot;/etc/containerd/certs.d&quot;

Here is the structure of /etc/containerd/certs.d

/etc/containerd/certs.d/
├── docker.io
│   └── hosts.toml
└── dsaber.repo.farbeen.local
    ├── ca.crt
    └── hosts.toml

And here is the hosts.toml content:

server = "http://dsaber.repo.farbeen.local"
[host."http://dsaber.repo.farbeen.local"]
  skip_verify = true
  override_path = false

From the node itself I am able to use ctr to pull the image using this command:

ctr --debug i pull dsaber.repo.farbeen.local/sad-rpc:0.0.1 --hosts-dir "/etc/containerd/certs.d"

And its output:

dsaber.repo.farbeen.local/sad-rpc:0.0.1:                                          resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:6d18e8ff88048b4188352066be4d940a6a3136600e2d6889dc5dfe0031496fd4: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:386ac04f537b2b7fcd5d34d2460cdfc8d1d45fe3bfdc370cd177923a4ae8785c:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:9b0ee25784e4f2b84ea0316526f8a7354afd0686862a60fc8c7896584b17b671:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:8f8f0e29dc664f8a844cdde94b51a3570546b749f300307a93b6024fa1d53950:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:7be92ad65ca3b889b0345f19ea1259baa85814bd2e24d746628a813a91e98e32:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:05ca1eb4c21a4e27a585b24cacbc09a91bbf06f0e29c1a65f08901f447d3897e:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:4640cee89851e78489135d92d62c478b1c64b077bf960c1b311cd0752196c7a2:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:44fa9f5c8769c8269fe13d76b1ebfe97b785d8e4bbb1877e5ef9310d24ba14cb:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:7ef3acb059e5af8a6df3d781a7ce7ca073e5b3b083a602a3b591631b5d93fb77:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e4fff0779e6ddd22366469f08626c3ab1884b5cbe1719b26da238c95f247b305:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:a81861217e67b37b54a16843d072ebbddb7c21e30ea480f8f295c1622e51946f:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:b0612a16bf22879c81d57fdd9e65b745cf74d57fea0ef7aa407b97b8e990e2ae:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:b40828aa8e1c54bfe18d573d45b9d55fcff7a7c2865697095f68e78afd92437b:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.1 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:6d18e8ff88048b4188352066be4d940a6a3136600e2d6889dc5dfe0031496fd4...
done: 22.837929ms

But when I want to pull the image using the following command:

crictl  pull dsaber.repo.farbeen.local/sad-rpc:0.0.1

It gives me this error:

crictl --debug  pull dsaber.repo.farbeen.local/sad-rpc:0.0.1
DEBU[0000] get image connection

DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:dsaber.repo.farbeen.local/sad-rpc:0.0.1,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},Auth:nil,SandboxConfig:nil,} 
E0820 11:54:45.096099   71166 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image &quot;dsaber.repo.farbeen.local/sad-rpc:0.0.1&quot;: failed to resolve reference &quot;dsaber.repo.farbeen.local/sad-rpc:0.0.1&quot;: unexpected status from HEAD request to http://dsaber.repo.farbeen.local/v2/sad-rpc/manifests/0.0.1: 502 Server dropped connection" image="dsaber.repo.farbeen.local/sad-rpc:0.0.1"
FATA[0007] pulling image: failed to pull and unpack image "dsaber.repo.farbeen.local/sad-rpc:0.0.1": failed to resolve reference "dsaber.repo.farbeen.local/sad-rpc:0.0.1": unexpected status from HEAD request to http://dsaber.repo.farbeen.local/v2/sad-rpc/manifests/0.0.1: 502 Server dropped connection

I can pull the image using nertctl also successfully

kubernetes fails to pull image

0 Answers0