How to make k8s imagePullPolicy = never work?

Question

I have followed the instructions on this blog to create a simple container image and deploy it in a k8s cluster.

However, in my case the pods do not run:

student@master:~$ k get pod -o wide -l app=hello-python --field-selector spec.nodeName=master
NAME                            READY   STATUS              RESTARTS   AGE     IP                NODE     NOMINATED NODE   READINESS GATES
hello-python-58547cf485-7l8dg   0/1     ErrImageNeverPull   0          2m26s   192.168.219.126   master   <none>           <none>
hello-python-598c594dc5-4c9zd   0/1     ErrImageNeverPull   0          2m26s   192.168.219.67    master   <none>           <none>
student@master:~$ sudo podman images hello-python
REPOSITORY              TAG         IMAGE ID      CREATED         SIZE
localhost/hello-python  latest      11cf1e5a86b1  50 minutes ago  941 MB
student@master:~$ hostname
master
student@master:~$

I understand why it may not work on the worker node, but why it does not work on the same node where the image is cached - the master node?

student@master:~$ k describe pod hello-python-58547cf485-7l8dg | grep -A 10 'Events:'
Events:
  Type     Reason             Age                   From               Message
  ----     ------             ----                  ----               -------
  Normal   Scheduled          10m                   default-scheduler  Successfully assigned default/hello-python-58547cf485-7l8dg to master
  Warning  Failed             8m7s (x12 over 10m)   kubelet            Error: ErrImageNeverPull
  Warning  ErrImageNeverPull  4m59s (x27 over 10m)  kubelet            Container image "localhost/hello-python:latest" is not present with pull policy of Never
student@master:~$

My question is: how to make the pod run on the master node with the imagePullPolicy = never given that the image in question is available on the master node as the podman images attests?

EDIT 1

I am using a k8s cluster running on two VMs deployed in GCE. It was setup with a script provided in the context of the Linux Foundation Kubernetes Developer course LFD0259.

EDIT 2

The master node is allowed to run workloads - this is how the LFD259 course sets it up. For example:

student@master:~$ k create deployment xyz --image=httpd
deployment.apps/xyz created
student@master:~$ k get pod -o wide
NAME                   READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
xyz-6c6bd4cd89-qn4zr   1/1     Running   0          5m37s   192.168.171.66   worker   <none>           <none>
student@master:~$
student@master:~$ k scale deployment xyz --replicas=10
deployment.apps/xyz scaled
student@master:~$ k get pod -o wide
NAME                   READY   STATUS              RESTARTS   AGE     IP                NODE     NOMINATED NODE   READINESS GATES
xyz-6c6bd4cd89-c2xv4   1/1     Running             0          73s     192.168.219.71    master   <none>           <none>
xyz-6c6bd4cd89-g89k2   0/1     ContainerCreating   0          73s     <none>            master   <none>           <none>
xyz-6c6bd4cd89-jfftl   0/1     ContainerCreating   0          73s     <none>            worker   <none>           <none>
xyz-6c6bd4cd89-kbdnq   1/1     Running             0          73s     192.168.219.106   master   <none>           <none>
xyz-6c6bd4cd89-nm6rt   0/1     ContainerCreating   0          73s     <none>            worker   <none>           <none>
xyz-6c6bd4cd89-qn4zr   1/1     Running             0          7m22s   192.168.171.66    worker   <none>           <none>
xyz-6c6bd4cd89-vts6x   1/1     Running             0          73s     192.168.171.84    worker   <none>           <none>
xyz-6c6bd4cd89-wd2ls   1/1     Running             0          73s     192.168.171.127   worker   <none>           <none>
xyz-6c6bd4cd89-wv4jn   0/1     ContainerCreating   0          73s     <none>            worker   <none>           <none>
xyz-6c6bd4cd89-xvtlm   0/1     ContainerCreating   0          73s     <none>            master   <none>           <none>
student@master:~$

you might need to load image, `eval $(minikube podman-env)` and then build the image `podman build -t my_image .` should be available in your cluster. https://minikube.sigs.k8s.io/docs/handbook/pushing/#3-pushing-directly-to-in-cluster-cri-o-podman-env — Adiii, Sep 29 '22 at 18:49
@Adiii - I edited my post - see **EDIT 1**. I am not running minikube. — mark, Sep 29 '22 at 18:55
The only way images move between places is by being pulled from a registry; Kubernetes won't copy images from the master to a worker (there probably aren't very many images on the master at all). You might find setting up a registry to be a more sustainable approach than manually copying images to every node in the cluster. — David Maze, Sep 29 '22 at 20:33
@DavidMaze - I am aware of that. This is why in my post I said I did not care about the worker node. I am asking specifically about the master node. This is the node where the image is present. I am going to emphasize it strongly in the post to avoid more confusion. — mark, Sep 29 '22 at 21:25
@mark I'm sorry but I've read your post 3 times now; and I still don't see/get what you're asking or wanting to achieve. Are you asking why the worker node can't pull the Image or do you want to prohibit the pull of the image in the first place? — Mike, Sep 30 '22 at 06:56
@MikeKilic - I have edited the post to emphasize my question. Is it better now or should I clarify more? Thank you. — mark, Sep 30 '22 at 13:47
@mark Is your issue resolved? If yes, can you provide the resolution steps you have followed and provide it as an answer for the greater visibility of the community? — Fariya Rahmat, Nov 24 '22 at 14:39
Please refer to the stack [link1](https://stackoverflow.com/questions/71248716/kuberneets-pods-images-errimageneverpull) [link2](https://stackoverflow.com/questions/36874880/kubernetes-cannot-pull-local-image) which may help to know the cause of the error. — Fariya Rahmat, Nov 25 '22 at 10:33

score 0 · Answer 1 · answered Oct 05 '22 at 10:45

0

It depends how you've set up your Kubernetes Cluster. I assume you've installed it with kubeadm. However, by default the Master is not scheduleable for workloads. And by my understanding the image you're talking about only exists on the master node right? If that's the case you can't start a pod with that Image as it only exists on the master node, which doesn't allow workloads by default.

If you were to copy the Image to the worker node, your given command should work.

However if you want to make your Master-Node scheduleable just taint it with (maybe you need to amend the last bit if it differs from yours):

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

answered Oct 05 '22 at 10:45

Mike

401
5

Hi. I did not think to mention it, but you are right, of course - I should have. The course sets up the cluster so that master can run workloads too. I have added **EDIT 2** to clarify this point. – mark Oct 05 '22 at 15:25
Thats rather odd then as I can see by the output that some pods are running when you scaled the deployment to 10 units. Is there any relevant info in the events/get/describe on those pods that have the status `ContainerCreating` – Mike Oct 10 '22 at 08:42
The `ContainerCreating` was just temporary. At the end all of the pods were running without restarts. The master node accepts workloads, this is the explicit decision on behalf of the course authors to minimize the number of GCE VMs for the cluster. – mark Oct 10 '22 at 14:42

How to make k8s imagePullPolicy = never work?

1 Answers1