10

I'm trying to run docker on windows (OS: Microsoft Windows 10 Pro 64bit, Docker ver: 18.09.0, build 4d60db4), by following the hello-world instruction here. Then I got this following "server misbehaving" error:

Error response from daemon: Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup <companyProxy> on 192.168.65.1:53: server misbehaving.

I tried to change the DNS (in Docker setting - Network - DNS Server) from Automatic to Fixed (8.8.8.8 or 8.8.4.4) as suggested here, but still did not solve the problem, and resulted in another type of error ("timeout exceeded").

Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).

I am behind company's proxy, and have set the proxy (and the credentials) both on environment variable and also docker setting. I also tried to reinstall both docker and hyperV but still got the same problem.

Can anybody help? Thanks

Leonard AB
  • 1,479
  • 1
  • 19
  • 30
  • I met a similar problem since yesterday under ubuntu. Docker pull always returned: 'Error response from daemon: Get https://registry-1.docker.io/v2/library/redis/manifests/latest: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fredis%3Apull&service=registry.docker.io: net/http: TLS handshake timeout' – Leo.W Nov 26 '18 at 17:26
  • Docker login always returned: 'Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)' – Leo.W Nov 26 '18 at 17:26
  • @Leo.W are you also behind company proxy? You said it was since yesterday, did it work before that? – Leonard AB Nov 26 '18 at 23:51
  • Yes and Yes to your questions. I have solved the problem and wrote an article about it. As you might be interested in, I posted an answer to this question with a link to the article. – Leo.W Dec 05 '18 at 18:33
  • Also ensure your proxy password doesn't contain special characters, especially `@` – KrishPrabakar Feb 11 '19 at 08:56

4 Answers4

8

We had this problem on Linux behind a corporate proxy after upgrading Docker from version 17 to the latest 19 (currently 19.03.5). 

# docker run hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup http on 1.2.3.4:53: server misbehaving.

1.2.3.4 is the IP of our DNS server, which itself worked fine - I could resolve different hosts, also registry-1.docker.io from Docker.

Solution

The problem was how we set the proxy globally in /etc/systemd/system/docker.service.d/http-proxy.conf. Since its an MS AD user, it contains the username in the format of domain\user like this:

[Service]
Environment="HTTP_PROXY=http://domain\user:password@proxyserver.internal:80"

Same thing for HTTPS_PROXY. While this worked on version 17, it doesn't seem to work with 19. Now the backslash seems to cause problems. Just remove it like this:

[Service]
Environment="HTTP_PROXY=http://user:password@proxyserver.internal:80"

How to check if this is a problem

I'm not sure if this changed with version 19 or already in version 18, since we skipped 18. But if you upgrade to 18 or 19 this is a thing i'd check. There is a simply way to figure it out: 

 docker info | grep -i proxy

If you see censored credentials like this

HTTP Proxy: http://xxxxx:xxxxx@proxyserver.internal:80
HTTPS Proxy: http://xxxxx:xxxxx@proxyserver.internal:80

then you're not affected of this issue. But if you see the plain credentials, Docker can't parse them because of the backslash or maybe other special characters included in your env variable.

Lion
  • 16,606
  • 23
  • 86
  • 148
  • Removing the domain from Environment="HTTP_PROXY=http: //domain\user:password@proxyserver.internal:80" worked for us. Thank you! – Anand Nov 26 '20 at 17:54
2

The timeout in your last configuration is caused by the fact that you're not getting through the proxy to those external DNS servers (8.8.8.8 - 8.8.4.4), I think.

You should be solving the first issue, being the server misbehaving. As was the case with me, mentioned here, this was caused by the unability of docker to authenticate with the proxy. My solution was to use cntml.

The cntlm configuration is actually really straightforward if you follow their guidelines. When you have cntlm running, you need to configure docker to use that as a proxy instead of your corporate proxy. Just a plain proxy without authentication - most likely 127.0.0.1:3128, if you're running it on the same machine. cntlm will handle the authentication on the corporate proxy.

As a reference, this is the cntlm config I used:

Username        <username>
Domain          <domain>
Password        <password>

PassLM          <PassLM output of cntlm -H>
PassNT          <PassNT output of cntlm -H>
PassNTLMv2      <PassNTLMv2 output of cntlm -H>

Proxy           <corporate proxy>:<corporate proxy port>
NoProxy         localhost, 127.0.0.*, 10.*, 192.168.*, 172.16.*.*

Listen          3128

You get the hashes by running cntlm -H -u <username>@<domain>.

Make sure you run cntlm in gateway mode cntlm -g (instead of cntlm -v).

Good luck!

Sylvain Girard
  • 368
  • 3
  • 13
  • I tried to install cntlm. When I run it, it said that port `3128` is already in use. Then I tried to use other port (I tried `netstat` as suggested, to find unused port), in this case I tried `8080`. Running `cntlm -v` showed no problem, but when I ran `docker run hello-world`, I got `Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp 10.0.75.1:8080: connect: connection refused.` Do you have any idea? – Leonard AB Nov 27 '18 at 08:06
  • 1
    I finally got it worked! the last missing piece was running cntlm in gateway mode `cntlm -g` (instead of `cntlm -v`). Thank you!!! – Leonard AB Nov 27 '18 at 08:24
  • did you manage to do `docker image build` with this setup? in my case, `docker run` (or `pull`, or `search`) worked fine, but when building and whenever it needs to access the internet (to run some command like `pip` or `apk`), it fails. I followed this solution https://stackoverflow.com/a/38901128/5122657 but still doesn't work – Leonard AB Nov 29 '18 at 07:16
  • No, I'm deploying built images from a remote repo. I suggest looking at some of these resources: https://stackoverflow.com/questions/48277599/pip-inside-dockerfile-under-proxy https://github.com/gliderlabs/docker-alpine/issues/171 https://github.com/gliderlabs/docker-alpine/issues/191 – Sylvain Girard Nov 29 '18 at 11:16
  • I'll take a look. Thanks! – Leonard AB Nov 29 '18 at 22:43
  • additional info: you can set the `Gateway` inside the `cntlm.ini` file into `true` so you don't need to set it every time – Leonard AB May 21 '19 at 00:19
0

I have accepted the above answer but, for an unknown reason, the problem reappeared after a few days with a slightly different error. Luckily I managed to solve it using a different setup, as written below. Hope it can help someone when the accepted answer does not work.

First of all, the condition where this error happens: CNTLM listens to 127.0.0.1:3128, docker proxy (set through the GUI) is 127.0.0.1:3128 both for HTTP and HTTPS

Error response from daemon: Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp 10.0.75.1:3128: connect: connection refused

I checked this similar case but the solution also did not work for me. I tried several different things including turning off firewall, and reproducing the initial problem (my initial question) and redoing the accepted solution, to no avail. I realized that the IP address shown in the error is not my corporate proxy, nor the Cntlm-listened port, nor my localhost, but the IP of the dockerNAT.

Then, I also noticed that when running CNTLM -g -v, below the Cntlm ready, staying in the foreground line, nothing showed up when I run docker pull hello-world (in another terminal).

Inspired by this answer, I tried to change the Cntlm.ini and the proxy setting on docker to 10.0.75.1:3128, and then restarted cntlm and docker. Now things changed slightly. The error message changed to this:

Error response from daemon: Get https://registry-1.docker.io/v2/: Parent proxy unreacheable

and Cntlm terminal now showed something when I hit the docker pull command, indicating that it somehow works. I suspected that the proxy hostname might cause the problem at this stage, since docker might not be able to access DNS server from the VM. Then I changed Cntlm.ini again, changing the hostname of the proxy into its IP address, and voila, now docker pull hello-world works normally!

(TBH I don't fully understand the network theory behind why this solution works. If anyone can give some explanation, that would be helpful.)

Leonard AB
  • 1,479
  • 1
  • 19
  • 30
0

I met the same problem with Windows 10 (Host OS) + VMware + Ubuntu.

In my case, the problem is caused by the company's firewall.

Just in case you might meet the same problem. I wrote an article about it:

Solve: Docker pull - "​... TLS handshake timeout"

Leo.W
  • 539
  • 1
  • 7
  • 18