private docker registry for gitlab secret not working in kubernetes

Question

I am trying to setup a gitlab private registry for my kubernetes container images.

I've cut the irrelevant code out below.

My replica set is defined as:

kind: ReplicaSet
...
    spec:
      containers:
      - name: redacted
        image: registry.gitlab.com/redacted/redacted/redacted:latest
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: redacted-data
          mountPath: /var/www/html
      imagePullSecrets:
      - name: github-auth
...

I'm setting my secret with the following kubectl command:

kubectl create -n redacted secret docker-registry gitlab-auth \
--docker-server="registry.gitlab.com:5000" \
--docker-username="redacted" \
--docker-password="redacted" \
--docker-email="redacted" \
--namespace="redacted"

Here is the failing container output:

Name:             redacted-cgbrk
...
Containers:
  redacted:
    Container ID:   
    Image:          registry.gitlab.com/redacted/redacted/redacted:latest
    Image ID:       
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qv24l (ro)
      /var/www/html from redacted-data (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True
...
Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        64s                default-scheduler        0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled               62s                default-scheduler        Successfully assigned redacted/redacted-cgbrk to pool-2t9lbcb5l-7d37n
  Normal   SuccessfulAttachVolume  55s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-6c4aac85-bb60-44e8-b557-7f65d62543fa"
  Normal   Pulling                 16s (x3 over 54s)  kubelet                  Pulling image "registry.gitlab.com/redacted/mpro/redacted:latest"
  Warning  Failed                  16s (x3 over 54s)  kubelet                  Failed to pull image "registry.gitlab.com/redacted/redacted/redacted:latest": rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/redacted/redacted/redacted:latest": failed to resolve reference "registry.gitlab.com/redacted/redacted/redacted:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
...

Answer 1

Kubernetes uses a separate auth than docker login, Check you may have configured Kubernetes with the required authentication so it can pull from your private registries.

Follow below steps :

1)Log in to Docker Hub

2)Create a Secret based on existing credentials

3)Create a Secret by providing credentials on the command line

4)Inspecting the Secret regcred

5)Create a Pod that uses your Secret

Please see K8S issue here: Pull an Image from a Private Registry for more information.

Also Refer to this Similar SO for more information.