Docker container can't connect to host Linux even when using host.docker.internal as an extra-host

Question

I want to be able to host a Prometheus instance in Docker, and have that instance get the host metrics using Prometheus Node Exporter.

To connect to the host from within a container, it should be possible to use the special DNS name host.docker.internal. The docs don't mention this, but apparently it is required to set this manually via extra_hosts on Linux - which I have done, but it still does not work.

Steps to reproduce

I first start listening on 0.0.0.0:55555 with ncat.

➜ ncat -vv -k -l -p 55555
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Listening on :::55555
Ncat: Listening on 0.0.0.0:55555

I then verify that, from the host machine, I can connect to ncat successfully:

➜ telnet localhost 55555
Trying ::1...
Connected to localhost.

➜ ncat -vv -k -l -p 55555
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Listening on :::55555
Ncat: Listening on 0.0.0.0:55555
Ncat: Connection from ::1.
Ncat: Connection from ::1:48416.

I then start the container using Docker Compose. As you can see, I have addedhost.docker.internal:host-gateway to extra_hosts, as directed this answer.

➜ docker-compose down; docker-compose up -d; docker-compose logs -f

version: "3.9"

services:
  prometheus:
    image: prom/prometheus:v2.42.0
    user: "1000:1000"
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    extra_hosts:
      host.docker.internal: "host-gateway"
    volumes:
      - "./config/prometheus.yml:/etc/prometheus/prometheus.yml"
        # Note that Prometheus requires the mounted `data/` directory 
        # is created with fully open permissions, `chmod -R 777 data`
      - "./data/prometheus:/prometheus"

I then open a terminal inside the container.

➜ docker exec -it --user root prometheus /bin/sh

And then I try to connect to the host ncat, but I get an error.

➜ telnet host.docker.internal 55555

telnet: can't connect to remote host (172.17.0.1): Connection refused

I have tried the following answers, which say I should add host.docker.internal to extra_hosts, which I have done, to no effect:

• https://stackoverflow.com/a/73859760/4161471
• What is linux equivalent of "host.docker.internal"
• https://stackoverflow.com/a/43541732/4161471

Debugging

I have tried a simpler Docker container example, based on the Docker docs, but it stil fails.

➜ docker run --add-host=host.docker.internal:host-gateway --rm -it alpine sh

# inside the container:
➜ apk add curl
➜ curl http://host.docker.internal:55555
# curl: (7) Failed to connect to host.docker.internal port 55555 after 0 ms: Couldn't connect to server

I am sure that the ncat server is connectable on tcp and tcp6, and listens to 0.0.0.0

➜ sudo netstat -tulpn | grep :55555
tcp        0      0 0.0.0.0:55555           0.0.0.0:*               LISTEN      16150/ncat
tcp6       0      0 :::55555                :::*                    LISTEN      16150/ncat

When I try pinging the host from within the container, it works (or, at least it does not throw an error)

➜ ping host.docker.internal:55555
PING host.docker.internal:55555 (172.17.0.1): 56 data bytes
64 bytes from 172.17.0.1: seq=0 ttl=64 time=0.049 ms
64 bytes from 172.17.0.1: seq=1 ttl=64 time=0.158 ms
^C
--- host.docker.internal:55555 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.049/0.103/0.158 ms

From within the container I can see that /etc/hosts has an entry for host.docker.internal

➜ cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.1      host.docker.internal
172.17.0.2      84a18e1e361b

Answer 1

I had the same problem, and it was iptables service on the host side. Whith iptables service stopped, the connection succeeded, so I just inserted a configuration on iptables to allow the connection to the docker metrics port and restarted it again.