MicroK8s does not authenticate with gitlab private registry

Question

I have an instance of MicroK8s, I am currently migrating my applications. But I can't authenticate to Gitlab's private registry.

About my environment:

MicroK8s
Gitlab (with registry working securely)

Tests I performed

To test my private Gitlab registry I used another machine and did a docker pull registry.mydomain.com/my-group/my-image:my-tag.

In K8s I followed the instructions available on the K8s website and also the instructions in the answer to this question on SO.

I've already checked the namespaces, the secret was created correctly! I restored the credentials to confirm and they are correct.

Different ways I used to create the secret for imagePullSecrets.

First try

kubectl create secret docker-registry regcred \
  --namespace=my-namespace \
  --docker-server=registry.mydomain.com \
  --docker-username="my-user" \
  --docker-email="my-mail" \
  --docker-password="my-gitlab-personal-access-token"

Second attempt

kubectl create secret generic regcred \
 --namespace=my-namespace \
 --from-file=.dockerconfigjson=/home/hsouza/.docker/config.json \
 --type=kubernetes.io/dockerconfigjson

Both ways of creating my secret work, however when creating my pod the ImagePullBackOff error is presented.

For both attempts to generate the regcred I created the POD like this:

apiVersion: v1
kind: Pod
metadata:
  namespace: my-namespace
  name: test-private-register-pod
spec:
  containers:
  - name: test-private-register-pod
    image: registry.mydomain.com/my-group/my-image:1.0.0
  imagePullSecrets:
  - name: regcred

When analyzing with kubectl describe pod/test-private-register-pod the following message is returned:

Name:             test-private-register-pod
Namespace:        my-namespace
Priority:         0
Service Account:  default
Node:             k8s-dev/10.0.0.6
Start Time:       Sat, 07 Oct 2023 11:25:23 -0300
Labels:           <none>
Annotations:      cni.projectcalico.org/containerID: 0d1a8c2d3f54ea9dfeccb69b96f930a6c5f40d6c9fa8c16994ac24676cecb5be
                  cni.projectcalico.org/podIP: 10.1.252.205/32
                  cni.projectcalico.org/podIPs: 10.1.252.205/32
Status:           Pending
IP:               10.1.252.205
IPs:
  IP:  10.1.252.205
Containers:
  test-private-register-pod:
    Container ID:   
    Image:          registry.mydomain.com/my-group/my-image:1.0.0
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8gj47 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-8gj47:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  16s                default-scheduler  Successfully assigned my-namespace/test-private-register-pod to k8s-dev
  Normal   Pulling    15s                kubelet            Pulling image "registry.mydomain.com/my-group/my-image:1.0.0"
  Warning  Failed     15s                kubelet            Failed to pull image "registry.mydomain.com/my-group/my-image:1.0.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mydomain.com/my-group/my-image:1.0.0": failed to resolve reference "registry.mydomain.com/my-group/my-image:1.0.0": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden
  Warning  Failed     15s                kubelet            Error: ErrImagePull
  Normal   BackOff    14s (x2 over 15s)  kubelet            Back-off pulling image "registry.mydomain.com/my-group/my-image:1.0.0"
  Warning  Failed     14s (x2 over 15s)  kubelet            Error: ImagePullBackOff

In the gitlab nginx logs, the status code for K8s access is 401. I put it in a fictitious way to try to pull from https://requestcatcher.com/ and when analyzing the request made by K8s it does not send the header with Basic Auth .

This is my problem, I appreciate the community's help in trying to identify where I'm going wrong.

MicroK8s does not authenticate with gitlab private registry

0 Answers0