When using Docker, ESTABLISHED connections don't appear in netstat

Question

I have a docker container running on RHEL 7 with Docker 1.7.0. The program which running in this container listens for TCP connections on port 5000. In my Dockerfile I put the sentence EXPOSE 5000 and I run the container with the following command:

docker run \
--name myProgram \
--detach \
--publish 5000:5000 \
--volume /home/docker/apps/myProgram/logs:/var/log/myProgram/ \
--volume /home/docker/apps/myProgram/conf:/usr/local/snnotificationreceiver/conf/ \
--restart always \
10.167.152.15:5000/myProgram:1.0.0

When I execute netstat on the host I see the LISTEN socket:

[root@server bin]# netstat -naop | grep 5000
tcp6       0      0 :::5000                 :::*                    LISTEN      33595/docker-proxy   off (0.00/0/0)

I can connect to the application by connecting to the host ip address on port 5000 and the data I send to the application arrives. I know this because I see it on my application logs, the application also sends data through the socket. However I don't see any ESTABLISHED connections using netstat on the docker host:

[root@server bin]# netstat -naop | grep ESTABLISHED

I see the ESTABLISHED connection on the client side which doesn't use docker:

[root@client ~]# netstat -naop | grep 5000
tcp        0      0 10.167.43.73:39218      10.167.152.138:5000     ESTABLISHED 21429/telnet         off (0.00/0/0)

I didn't find any docker command equivalent or similar to netstat Is this normal? How can I see the ESTABLISHED connections to a container or to the docker-proxy?

Thanks

AdvilUser · Accepted Answer · 2016-11-16T17:29:29.063

29

You can either do:

docker exec <containerid> netstat -tan | grep ESTABLISHED

or if you don't have netstat in your docker image:

docker inspect -f '{{.State.Pid}}' <containerid> # note the PID
sudo nsenter -t <pid> -n netstat | grep ESTABLISHED

nsenter is part of util-linux package. (plagiarized @larsks)

edited Nov 16 '16 at 17:29

answered Nov 01 '16 at 03:46

AdvilUser

3,142
3
25
15

2

Actually it's a working solution, but the command should looks like `sudo nsenter -t -n netstat | grep ESTABLISHED`. Pay attention to the `-n` option. More information can be found here http://stackoverflow.com/a/40352004/1201488. – Ivan Velichko Nov 02 '16 at 22:30
1

Yeah sorry about that. -n is required for nsenter to work here. – AdvilUser Nov 16 '16 at 17:30
To me this shows connections as coming from localhost, no clue about the source address of the connection. Anybody knows how to see that? – jjmontes Nov 18 '16 at 09:47
2

Using 'ss' instead of 'netstat' can be preferable. You can specify an optional filter and see both ends of the connection. Example input: ```nsenter -t 73647 -n ss -pnt state established 'dport = 3128' dst 10.1.2.3``` – Cameron Kerr Oct 31 '17 at 09:33
Just for convenience. If those docker container resources are orchestrated and managed by a Kubernetes depoyment object (Replica Sets), they will have a label `io.kubernetes.container.name={.spec.template.spec.containers[*].name}` (label value is expressed in jsonpath syntax and can be viewed from the CONTAINERS column by `kubectl get deployment -owide`). To get specified container name/id, use `docker container ls -f label=io.kubernetes.container.name=my-nginx --format={{.Names}}` or `docker container ls -f label=io.kubernetes.container.name=my-nginx --format={{.ID}}`. – samm Aug 12 '20 at 11:15

score 6 · Answer 2 · edited Oct 10 '19 at 13:37

You may use this snippet to get all the ESTABLISHED for all dockers in one row (if you got nsenter)

docker inspect --format '{{.State.Pid}} {{printf "%.13s" .ID}} {{.Name}}' \
$(docker ps --format '{{.ID}}') | while read dockpid dockid dockname
    do
    echo $dockid $dockname
    sudo nsenter -t $dockpid -n netstat -pan | grep ESTABLISHED
done

note the ESTABLISHED in the grep.

you can change to the listening connection with netstat -pnl both TCP and UDP

docker inspect --format '{{.State.Pid}} {{printf "%.13s" .ID}} {{.Name}}' \
$(docker ps --format '{{.ID}}') | while read dockpid dockid dockname
    do
    echo $dockid $dockname
    sudo nsenter -t $dockpid -n netstat -pnl
done

or only TCP LISTEN

docker inspect --format '{{.State.Pid}} {{printf "%.13s" .ID}} {{.Name}}' \
$(docker ps --format '{{.ID}}') | while read dockpid dockid dockname
    do
    echo $dockid $dockname
    sudo nsenter -t $dockpid -n netstat -pnlt
done

Should be `echo $dockid $dockname` I think, as it uses the variables set by the preceding `read dockpid dockid dockname`. — MSalters, Sep 18 '18 at 10:10

When using Docker, ESTABLISHED connections don't appear in netstat

2 Answers2

Linked