1

I am trying installing a simple Spring Boot application by JKube on a microk8s cluster. As remote private docker repository I am using GitHub Packages (beta). While the microk8s is getting the pod, he unsuccessfully tries to deploy it and brings out following error:

Command:

kubectl describe pod walking-skeleton-5df7c4998c-9blmz -n walking

Output:

Name:         walking-skeleton-5df7c4998c-9blmz
Namespace:    walking
Priority:     0
Node:         server-k8s-main/MYIP
Start Time:   Tue, 16 Feb 2021 23:01:57 +0100
Labels:       app=walking-skeleton
              group=de.MYNAME
              pod-template-hash=5df7c4998c
              provider=jkube
              version=0.0.2-SNAPSHOT
Annotations:  cni.projectcalico.org/podIP: 10.1.4.29/32
              cni.projectcalico.org/podIPs: 10.1.4.29/32
              jkube.io/git-branch: main
              jkube.io/git-commit: c0ba4b997db326fc8a7018adfca4f5a4d74aea39
              jkube.io/git-url: git@github.com:MYNAME/MYPROJECT.git
              jkube.io/scm-tag: HEAD
              jkube.io/scm-url: https://github.com/spring-projects/spring-boot/walking-skeleton
Status:       Pending
IP:           10.1.4.29
IPs:
  IP:           10.1.4.29
Controlled By:  ReplicaSet/walking-skeleton-5df7c4998c
Containers:
  spring-boot:
    Container ID:   
    Image:          docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT
    Image ID:       
    Ports:          8080/TCP, 9779/TCP, 8778/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8080/actuator/health delay=180s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:8080/actuator/health delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      KUBERNETES_NAMESPACE:  walking (v1:metadata.namespace)
      HOSTNAME:              walking-skeleton-5df7c4998c-9blmz (v1:metadata.name)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jq7q9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-jq7q9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jq7q9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  13m                  default-scheduler  Successfully assigned walking/walking-skeleton-5df7c4998c-9blmz to server-k8s-main
  Normal   Pulling    11m (x4 over 13m)    kubelet            Pulling image "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT"
  Warning  Failed     11m (x4 over 13m)    kubelet            Error: ErrImagePull
  Warning  Failed     11m (x4 over 13m)    kubelet            Failed to pull image "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT": rpc error: code = Unknown desc = failed to pull and unpack image "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT": failed to resolve reference "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT": unexpected status code [manifests 0.0.2-SNAPSHOT]: **401 Unauthorized**
  Normal   BackOff    11m (x6 over 13m)    kubelet            Back-off pulling image "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT"
  Warning  Failed     3m1s (x43 over 13m)  kubelet            Error: ImagePullBackOff

I think, the interesting line is:

failed to resolve reference "docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:0.0.2-SNAPSHOT": unexpected status code [manifests 0.0.2-SNAPSHOT]: **401 Unauthorized**

My plugin config in the pom.xml looks like:

        <profile>
            <id>kubernetes-deploy</id>
            <properties>
                <jkube.docker.registry>docker.pkg.github.com/MYNAME/MYPROJECT</jkube.docker.registry>
                <jkube.generator.name>docker.pkg.github.com/MYNAME/MYPROJECT/walkingskeleton:${project.version}
                </jkube.generator.name>
            </properties>

            <build>
                <plugins>
                    <plugin>
                        <groupId>org.eclipse.jkube</groupId>
                        <artifactId>kubernetes-maven-plugin</artifactId>
                        <version>1.1.0</version>
                        <configuration>
                            <namespace>${k8s.namespace}</namespace>
                            <resources>
                                <replicas>2</replicas>
                                <secrets>
                                    <secret>
                                        <dockerServerId>docker.pkg.github.com/MYNAME/MYPROJECT</dockerServerId>
                                        <name>github</name>
                                        <namespace>${k8s.namespace}</namespace>
                                    </secret>
                                </secrets>
                            </resources>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>

The namespace is configured like <k8s.namespace>walking</k8s.namespace> in the properties. And as secret in the settings:

        <server>
            <id>docker.pkg.github.com/MYNAME/MYPROJECT</id>
            <username>MYNAME</username>
            <password>nice try :-)</password>
        </server>

I also checked, that the secret has been deployed on the microk8s:

kubectl get secrets -n walking

NAME                  TYPE                                  DATA   AGE
default-token-jq7q9   kubernetes.io/service-account-token   3      37m
github                kubernetes.io/dockercfg               1      37m

Can you help me? What I am missing? Thank you in advance :-)

  • How are you authenticating to the private repository? The error displayed on the describe says ErrImage pull and further down it says unauthorized which suggests the it is not able to pull the image from the repository – camba1 Feb 17 '21 at 03:55
  • I thought that was what the Kubernetes secret was for (github kubernetes.io/dockercfg). – Andreas Gp. Feb 17 '21 at 08:06
  • 1
    You need to provide an imagepullsecret in your yaml. Since you are using github registry, take a look at this article on how to authenticate https://dev.to/asizikov/using-github-container-registry-with-kubernetes-38fb – camba1 Feb 17 '21 at 16:23
  • It really looks like that the part "imagePullSecrets:- name: github" is missing. The deployment file is written by JKube automatically, so I patched it manually. Then the next error rises (nothing related to this story). Will have a look at the JKube issues. – Andreas Gp. Feb 17 '21 at 20:45
  • 1
    Hi, I'm from Eclipse JKube team. Were you able to resolve this issue? – Rohan Kumar Feb 19 '21 at 12:48
  • Hi Rohan, I let the kubernetes files generated by the maven goal "k8s:resources", copied them into /src/main/jkube and added the missing imagePullSecrets-Tag. I learned about your issue #171 at github and really has been surprised, that I can perfectly create a kubernetes secret by using resources/secrets/secret, but not for using it pulling images from a private docker image. I love jkube, also because using kubernetes in pipelines has been "heavy" before it and I can live with some workarounds ... but adding imagePullSecrets would help much. – Andreas Gp. Feb 19 '21 at 19:18
  • @AndreasGp.: Sorry for late reply. Thanks a lot for your feedback. I think `jkube.build.pullsecret` is specific to openshift only. One of our users who was using Jkube on top of Azure Kubernetes Service also seem to be facing the same issue was able to get it working via resource fragment: https://github.com/hanghan93/jkube-k8s-maven/tree/master/k8s-generate-resources-fragments – Rohan Kumar Mar 04 '21 at 13:43
  • @AndreasGp.: Please try it out and provide feedback. Feel free to contact us for any query on our gitter channel or Github issues: https://gitter.im/eclipse/jkube – Rohan Kumar Mar 04 '21 at 13:46
  • Thank you very much for this hint, I will try out (looks like my solution, but only have a fragment of the deployment descriptors manually created). I think it will make sense to create a feature request at jkube's github? – Andreas Gp. Mar 11 '21 at 10:38
  • polite ping, were you able to resolve this issue? – Rohan Kumar Apr 12 '21 at 06:28
  • @RohanKumar: this is a late response, but I stumbled upon this issue and found a solution for me. See my answer: https://stackoverflow.com/a/76056701/3344078 – tschale Apr 19 '23 at 16:11

1 Answers1

1

I've had the same issue when upgrading an existing cluster to K8s 1.24. The image of a deployment couldn't be loaded anymore due to an authentication issue.

The solution for me was to change the registry from docker.pkg.github.com to ghcr.io. This is mentioned in a containerd GitHub issue, which references the GitHub docs:

The domain in the pull URL will be ghcr.io instead of docker.pkg.github.com.

tschale
  • 975
  • 1
  • 18
  • 34