2

I have a local minikube setup using hyperkit VM. I installed helm on my system and the client was installed successfully but tiller is not installing in the cluster. The status for the pod is ImagePullBackOff

I tried installing different versions of the image with the --tiller-image flag but it still fails. If I do a normal docker pull of the same image in my system it works.

The command helm version gives this result:

Client: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}
Error: could not find a ready tiller pod

The command kubectl -n kube-system get pods gives this results:

NAME                               READY   STATUS             RESTARTS   AGE
coredns-fb8b8dccf-bh6w7            1/1     Running            0          146m
coredns-fb8b8dccf-gskc8            1/1     Running            1          146m
etcd-minikube                      1/1     Running            0          145m
kube-addon-manager-minikube        1/1     Running            0          145m
kube-apiserver-minikube            1/1     Running            0          145m
kube-controller-manager-minikube   1/1     Running            0          145m
kube-proxy-jqb9b                   1/1     Running            0          146m
kube-scheduler-minikube            1/1     Running            0          145m
storage-provisioner                1/1     Running            0          146m
tiller-deploy-548df79d66-xsngk     0/1     ImagePullBackOff   0          27m

The command kubectl -n kube-system describe pod tiller-deploy-548df79d66-xsngk fives the below result:

Name:               tiller-deploy-548df79d66-xsngk
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               minikube/192.168.64.56
Start Time:         Sat, 06 Jul 2019 22:41:54 +0530
Labels:             app=helm
                    name=tiller
                    pod-template-hash=548df79d66
Annotations:        <none>
Status:             Pending
IP:                 172.17.0.5
Controlled By:      ReplicaSet/tiller-deploy-548df79d66
Containers:
  tiller:
    Container ID:   
    Image:          gcr.io/kubernetes-helm/tiller:v2.12.1
    Image ID:       
    Ports:          44134/TCP, 44135/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:44135/liveness delay=1s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:44135/readiness delay=1s timeout=1s period=10s #success=1 #failure=3
    Environment:
      TILLER_NAMESPACE:    kube-system
      TILLER_HISTORY_MAX:  0
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6w54d (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-6w54d:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6w54d
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  21h                  default-scheduler  Successfully assigned kube-system/tiller-deploy-548df79d66-xsngk to minikube
  Warning  Failed     21h                  kubelet, minikube  Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.12.1": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:46818->192.168.64.1:53: read: connection refused
  Warning  Failed     21h                  kubelet, minikube  Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.12.1": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:32929->192.168.64.1:53: read: connection refused
  Warning  Failed     21h                  kubelet, minikube  Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.12.1": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:59446->192.168.64.1:53: read: connection refused
  Warning  Failed     21h (x4 over 21h)    kubelet, minikube  Error: ErrImagePull
  Normal   Pulling    21h (x4 over 21h)    kubelet, minikube  Pulling image "gcr.io/kubernetes-helm/tiller:v2.12.1"
  Warning  Failed     21h                  kubelet, minikube  Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.12.1": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:58643->192.168.64.1:53: read: connection refused
  Warning  Failed     21h (x7 over 21h)    kubelet, minikube  Error: ImagePullBackOff
  Normal   BackOff    21h (x110 over 21h)  kubelet, minikube  Back-off pulling image "gcr.io/kubernetes-helm/tiller:v2.12.1"

I am not behind any proxy if that helps.

Update 1:

I ssh-ed into minikube and tried to pull the docker image manually but I got the error:

Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:46912->192.168.64.1:53: read: connection refused

Update 2:

This issue is not only for just helm, but for any image. I usw Quay.io for images and the cluster can't reach that also, I tried kafka and that fails too. If I ssh into minikube, update resolv and then manually do a docker pull inside minikube, the pod spins up.

Yashu Gupta
  • 21
  • 2
  • 5

1 Answers1

1

Reuse the docker daemon for minikube with the command,

eval $(minikube docker-env)

and then pull the tiller image like this,

docker pull gcr.io/kubernetes-helm/tiller:v2.12.1

Now minikube will have the image downloaded. Delete the existing tiller pod with this command,

kubectl delete pod tiller-deploy-548df79d66-xsngk -n kube-system

The explanation is this. Since it looks like the ImagePullPolicy is not mentioned in the tiller deployment and also as the default ImagePullPolicy is IfNotPresent, minikube might not download the image and will try to use the already dowloaded image.

Update: The reason that helm was not able to install tiller, being that the resolv.conf file of minikube container had nameserver 192.168.64.1 from the previous network that he got connected to while creating the cluster. So minikube was not able to pull tiller docker image. It might probably be due to the fact that docker sets up the nameserver based on the network you are connected to.

This link may help.

Malathi
  • 2,119
  • 15
  • 40
  • 1
    The [default](https://kubernetes.io/docs/concepts/containers/images/#updating-images) `imagePullPolicy: IfNotPresent` will try to pull the image on its own, and you see that in the `kubectl describe` output in the question. It's possible this will help get past the immediate question, but they'll have to repeat this step for every single image they use, which isn't normal behavior. – David Maze Jul 07 '19 at 18:47
  • I tried piulling the image after doing the eval and I still have the same error: `Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 192.168.64.1:53: read udp 192.168.64.56:51347->192.168.64.1:53: read: connection refused` But weirdly, if I do a docker pull without evaling, it still works. – Yashu Gupta Jul 07 '19 at 19:08
  • Are you able to pull the docker image after you logged into minikube with minikube ssh ? Also how did you install tiller? – Malathi Jul 07 '19 at 19:09
  • I haven't tried after ssh-ing in minikube. And I didn't do anything special to install tiller. I just did helm init to install it. – Yashu Gupta Jul 07 '19 at 19:10
  • Nope, I can't pull after I ssh in minikube. Same error. – Yashu Gupta Jul 07 '19 at 19:12
  • Could there be a network issue? As in, the cluster was created from a different network than the one I am on right now. – Yashu Gupta Jul 07 '19 at 19:17
  • Can you see if internet is available within minikube? Probably cat the contents of `/etc/resolv.conf` – Malathi Jul 07 '19 at 19:18
  • I cat-ed the contents of resolv.conf and it has `nameserver 192.168.64.1`. – Yashu Gupta Jul 07 '19 at 19:24
  • Can you add `nameserver 8.8.8.8` to resolv.conf and try again? – Malathi Jul 07 '19 at 19:25
  • Ok. So I tried ping gcr.io and I'm not able to do that from inside minikube – Yashu Gupta Jul 07 '19 at 19:26
  • can you ssh to minikube and then run docker pull and see? – Malathi Jul 07 '19 at 19:27
  • Ok yes. I can do a docker pull after I added the nameserver. Any idea what the issue was here? Why this would have happened? – Yashu Gupta Jul 07 '19 at 19:29
  • I hope you will be able to install helm now. Did you create the cluster when you were in a different network and then switched to another network? Looks like docker was able to resolv dns name with `192.168.64.1` in the previous network and in your current network the dns server doesn't exist – Malathi Jul 07 '19 at 19:32
  • Yes, I can use helm again. Installed charts now. And yes I was in a different network. But what can I do to solve this issue permanently? Updating resolv manually doesn't seem like a permanent solution. Is it? – Yashu Gupta Jul 07 '19 at 19:34
  • Nope. `nameserver 8.8.8.8` is google dns server. It will work in all the networks. Aso you can add another nameserver in resolv.conf. – Malathi Jul 07 '19 at 19:36
  • 1
    Yeah I Know 8.8.8.8 is Google DNS server. But resolv.conf will get over written and the nameserver would again need to be updated. So is there any way that you know of that I can ensure this is updated automatically? – Yashu Gupta Jul 07 '19 at 19:38
  • Can you do a minikube restart and see if this issue persists? Also are you using the latest version of minikube? – Malathi Jul 07 '19 at 19:41