12

I'm trying to use Kubernetes with Docker. My image runs with Docker. I have one master-node and two worker-nodes. I also created a local registry like this $ docker run -d -p 5000:5000 --restart=always --name registry registry:2 and pushed my image into it. Everything worked fine so far.

I added { "insecure-registries":["xxx.xxx.xxx.xxx:5000"] } to the daemon.json file at /etc/docker. And I also changed the content of the docker-file at /etc/default/to DOCKER_OPTS="--config-file=/etc/docker/daemon.json". I made the changes on all nodes and I restarted the docker daemon afterwards.

I am able to pull my image from every node with the following command:

sudo docker pull xxx.xxx.xxx.xxx:5000/helloworldimage

I try to create my container from the master node with the command bellow:

sudo kubectl run test --image xxx.xxx.xxx.xxx:5000/helloworldimage

Than I get the following error:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  <unknown>          default-scheduler  Successfully assigned default/test-775f99f57-m9r4b to rpi-2
  Normal   BackOff    18s (x2 over 44s)  kubelet, rpi-2     Back-off pulling image "xxx.xxx.xxx.xxx:5000/helloworldimage"
  Warning  Failed     18s (x2 over 44s)  kubelet, rpi-2     Error: ImagePullBackOff
  Normal   Pulling    3s (x3 over 45s)   kubelet, rpi-2     Pulling image "xxx.xxx.xxx.xxx:5000/helloworldimage"
  Warning  Failed     3s (x3 over 45s)   kubelet, rpi-2     Failed to pull image "xxx.xxx.xxx.xxx:5000/helloworldimage": rpc error: code = Unknown desc = failed to pull and unpack image "xxx.xxx.xxx.xxx:5000/helloworldimage:latest": failed to resolve reference "xxx.xxx.xxx.xxx:5000/helloworldimage:latest": failed to do request: Head https://xxx.xxx.xxx.xxx:5000/v2/helloworldimage/manifests/latest: http: server gave HTTP response to HTTPS client
  Warning  Failed     3s (x3 over 45s)   kubelet, rpi-2     Error: ErrImagePull

This is the docker version I use:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:37:22 2019
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:31:17 2019
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

This is the Kubernetes version I use:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0+k3s.1", GitCommit:"0f644650f5d8e9f091629f860b342f221c46f6d7", GitTreeState:"clean", BuildDate:"2020-01-06T23:20:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0+k3s.1", GitCommit:"0f644650f5d8e9f091629f860b342f221c46f6d7", GitTreeState:"clean", BuildDate:"2020-01-06T23:20:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/arm"}```
angelika285
  • 255
  • 1
  • 3
  • 12
  • Have you restarted the docker daemon after adding `"insecure-registries"` option to daemon.json file? – Shashank V Jan 21 '20 at 15:58
  • @ShashankV Yes, I restarted the docker daemon afterwards. – angelika285 Jan 21 '20 at 16:03
  • Disregard, please, if that's the case already but the change to add `"insecure-registries"` should be on all cluster nodes - the master and two worker nodes the cluster has. – gears Jan 21 '20 at 17:12
  • @gears the change is on all cluster nodes – angelika285 Jan 22 '20 at 09:01
  • What are your `kubernetes` and `docker` versions ? *I am able to pull my image from every node with the following command...* - from all nodes including `master` ? – mario Jan 22 '20 at 11:59
  • @mario I added the versions to the description. Yes, I can pull it from all nodes including `master`. – angelika285 Jan 22 '20 at 12:17
  • @angelika285 what os do you use ? Did you use any specific tool for setting up your cluster like kubespray etc. ? – mario Jan 29 '20 at 12:07
  • I see that you're using some flavour of linux (`Platform:"linux/arm"`) so you may want to try [this](https://github.com/docker/distribution/issues/1874#issuecomment-287135486) solution, more in detail described [here](https://github.com/docker/distribution/issues/1874#issuecomment-293609804) – mario Jan 29 '20 at 12:20
  • Similar question has been asked here https://stackoverflow.com/questions/51893120/kubernetes-pull-from-insecure-docker-registry You may be better off making the registry secure. – Filip Nikolov Feb 14 '20 at 00:39
  • Can try solution described [here](https://github.com/distribution/distribution/issues/1874#issuecomment-468101614) and let me know if it works for you. – acid_fuji Feb 22 '21 at 09:12
  • I am having this exact same problem using kubeadm v 1.27 I can docker pull the images to all the worker nodes from the registry on the master node. But the k8s pod fails to pull the image with the exact same message you got. Did you ever resolve this? – Dan Jun 25 '23 at 04:37

2 Answers2

2

It appears that in some situations solution described here solved the problem:

  1. sudo systemctl edit docker
  2. Add below lines:
  • [Service]
  • ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry registry:5000
  1. sudo systemctl daemon-reload
  2. systemctl restart docker
  3. systemctl status docker
acid_fuji
  • 6,287
  • 7
  • 22
2

Kubernetes: Failed to pull image. Server gave HTTP response to HTTPS client.

{ "insecure-registries":["xxx.xxx.xxx.xxx:5000"] }

to the daemon.json file at /etc/docker.

I solved this problem by configuring it on all kubernetes nodes.

Nikola Malešević
  • 1,738
  • 6
  • 21
  • 44
Alexey Sabo
  • 21
  • 2