kubectl can't connect to docker registry to download image

Question

I'm stepping through Kubernetes in Action to get more than just familiarity with Kubernetes.

I already had a Docker Hub account that I've been using for Docker-specific experiments.

As described in chapter 2 of the book, I built the toy "kubia" image, and I was able to push it to Docker Hub. I verified this again by logging into Docker Hub and seeing the image.

I'm doing this on Centos7.

I then run the following to create the replication controller and pod running my image:

kubectl run kubia --image=davidmichaelkarr/kubia --port=8080 --generator=run/v1

I waited a while for statuses to change, but it never finishes downloading the image, when I describe the pod, I see something like this:

  Normal   Scheduled              24m                 default-scheduler  Successfully assigned kubia-25th5 to minikube
  Normal   SuccessfulMountVolume  24m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-x5nl4"
  Normal   Pulling                22m (x4 over 24m)   kubelet, minikube  pulling image "davidmichaelkarr/kubia"
  Warning  Failed                 22m (x4 over 24m)   kubelet, minikube  Failed to pull image "davidmichaelkarr/kubia": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

So I then constructed the following command:

curl -v -u 'davidmichaelkarr:**' 'https://registry-1.docker.io/v2/'

Which uses the same password I use for Docker Hub (they should be the same, right?).

This gives me the following:

* About to connect() to proxy *** port 8080 (#0)
*   Trying **.**.**.**...
* Connected to *** (**.**.**.**) port 8080 (#0)
* Establish HTTP proxy tunnel to registry-1.docker.io:443
* Server auth using Basic with user 'davidmichaelkarr'
> CONNECT registry-1.docker.io:443 HTTP/1.1
> Host: registry-1.docker.io:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.docker.io
*       start date: Aug 02 00:00:00 2017 GMT
*       expire date: Sep 02 12:00:00 2018 GMT
*       common name: *.docker.io
*       issuer: CN=Amazon,OU=Server CA 1B,O=Amazon,C=US
* Server auth using Basic with user 'davidmichaelkarr'
> GET /v2/ HTTP/1.1
> Authorization: Basic ***
> User-Agent: curl/7.29.0
> Host: registry-1.docker.io
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"
< Date: Wed, 24 Jan 2018 18:34:39 GMT
< Content-Length: 87
< Strict-Transport-Security: max-age=31536000
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host *** left intact

I don't understand why this is failing auth.

Update:

Based on the first answer and the info I got from this other question, I edited the description of the service account, adding the "imagePullSecrets" key, then I deleted the replicationcontroller again and recreated it. The result appeared to be identical.

This is the command I ran to create the secret:

kubectl create secret docker-registry regsecret --docker-server=registry-1.docker.io --docker-username=davidmichaelkarr --docker-password=** --docker-email=**

Then I obtained the yaml for the serviceaccount, added the key reference for the secret, then set that yaml as the settings for the serviceaccount.

This are the current settings for the service account:

$ kubectl get serviceaccount default -o yaml
apiVersion: v1
imagePullSecrets:
- name: regsecret
kind: ServiceAccount
metadata:
  creationTimestamp: 2018-01-24T00:05:01Z
  name: default
  namespace: default
  resourceVersion: "81492"
  selfLink: /api/v1/namespaces/default/serviceaccounts/default
  uid: 38e2882c-009a-11e8-bf43-080027ae527b
secrets:
- name: default-token-x5nl4

Here's the updated events list from the describe of the pod after doing this:

Events:
  Type     Reason                 Age               From               Message
  ----     ------                 ----              ----               -------
  Normal   Scheduled              7m                default-scheduler  Successfully assigned kubia-f56th to minikube
  Normal   SuccessfulMountVolume  7m                kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-x5nl4"
  Normal   Pulling                5m (x4 over 7m)   kubelet, minikube  pulling image "davidmichaelkarr/kubia"
  Warning  Failed                 5m (x4 over 7m)   kubelet, minikube  Failed to pull image "davidmichaelkarr/kubia": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   BackOff                4m (x6 over 7m)   kubelet, minikube  Back-off pulling image "davidmichaelkarr/kubia"
  Warning  FailedSync             2m (x18 over 7m)  kubelet, minikube  Error syncing pod

What else might I be doing wrong?

Update:

I think it's likely that all these issues with authentication are unrelated to the real issue. The key point is what I see in the pod description (breaking into multiple lines to make it easier to see):

Warning  Failed                 22m (x4 over 24m)   kubelet, 
minikube  Failed to pull image "davidmichaelkarr/kubia": rpc error: code = 
Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/:
net/http: request canceled while waiting for connection
(Client.Timeout exceeded while awaiting headers)

The last line seems like the most important piece of information at this point. It's not failing authentication, it's timing out the connection. In my experience, something like this is usually caused by issues getting through a firewall/proxy. We do have an internal proxy, and I have those environment variables set in my environment, but what about the "serviceaccount" that kubectl is using to make this connection? Do I have to somehow set a proxy configuration in the serviceaccount description?

Answer 1

You need to make sure the Docker daemon running in the Minikube VM uses your corporate proxy by starting minikube along these lines:

minikube start --docker-env http_proxy=http://proxy.corp.com:port --docker-env https_proxy=http://proxy.corp.com:port --docker-env no_proxy=192.168.99.0/24

Answer 2

I faced same issue couple of time. Updating here, might be useful for someone.

First describe the POD(kubectl describe pod <pod_name>),

1. If you see access denied/repository does not exist errors like

Error response from daemon: pull access denied for test/nginx, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Solution:

• If local K8s, you need to login into docker registry first OR
• if Kubernetes Cluster on Cloud, create secret for Registry and add imagepullsecret along with secret name

---

2. If you get timeout error,

Error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Solution:

• check the node is able to connect network OR able to reach private/public Registry.
• If AWS EKS Cluster, you need to enable auto-assign ip to Subnet where EC2 is running.

Answer 3

To fetch images stored on registries that require credentials, you need to create a special type of secret called imagePullSecrets.

kubectl create secret docker-registry regsecret --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

Then create the Pod specifying the imagePullSecrets field

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: regsecret

Answer 4

As mentioned in my comment to the original post, I had the same issue. The only thing of note is the minikube was up as at creation. I restarted the underlying VM and image pulls started working.

Answer 5

This seems to be quite old issue, but I have similar issue and solved by logged in to your docker account.

You can try it by deleting the existing failed pods, firing "docker login" command (login to your acc), then retry for the pod creation.