6

For some reason whenever I suspend my VM and resume it, I can no longer connect to the docker container that is hosted within the VM. Usually, I pass -p 3000:3000 to the docker container so that I can access the rails instance within it and this works fine, but when I suspend the VM and resume it later, I can no longer connect to port 3000 even though it's listening within the docker image.

This results in me having to reboot the VM as service docker restart does not change anything.

Is there something else I should be looking at to resolve this issue? I've been suspending/resuming my VM with docker in it for quite awhile and have never run into this issue before.

EDIT

To reproduce this issue, I simply resumed my VM and tried connecting to localhost port 3000 from the VM itself (not within the docker image) and it cannot connect. However, below shows that port 3000 is listening:

[root:kali:~/app]# curl http://localhost:3000
curl: (56) Recv failure: Connection reset by peer
[root:kali:~/app]# netstat -antp | grep -i listen
tcp        0      0 127.0.0.1:43050         0.0.0.0:*               LISTEN      84770/autossh       
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      20478/sshd          
tcp6       0      0 :::3000                 :::*                    LISTEN      32731/docker-proxy  
tcp6       0      0 :::3001                 :::*                    LISTEN      32715/docker-proxy  
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 :::22                   :::*                    LISTEN      20478/sshd

From within docker, I can see that rails is working:

[root:77f444beafff:~/app]# rails s --binding 0.0.0.0
=> Booting Puma
=> Rails 5.2.3 application starting in development 
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 3.12.1 (ruby 2.5.1-p57), codename: Llamas in Pajamas
* Min threads: 5, max threads: 5
* Environment: development
* Listening on tcp://0.0.0.0:3000
Use Ctrl-C to stop

And here's the netstat from within docker:

[root:77f444beafff:~/app]# netstat -antp | grep -i listen
tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      478/redis-server *: 
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      765/puma 3.12.1 (tc 
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::6379

If I curl from within the docker image, I can see it hits the rails app just fine:

[root:77f444beafff:~/app]# curl http://localhost:3000/ -I
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Content-Type: text/html; charset=utf-8
ETag: W/"5078d30a6c1a5f6fc5cb7f9a82cd89f5"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: _vspm_session=Cace%2FN0zB%2F6QJOiietbuHxTHOMZUMuRmEukYqQTNaHQ91hskaN%2BPJzev0KdGUAAtYx9a35Mqdkr8eRkPdH4qOl6vOaCcPU0gy8s7IMfkb9VhRGPPbecepmI%2F9leA2dnD694P8ctXSBklOCnjhN0%3D--SglWrWvx3BFEAI3z--IkylACdXbR6eF27Hgn0Cgg%3D%3D; path=/; HttpOnly
X-Request-Id: 29aa7251-f29a-4309-adec-6af479e7bd9b
X-Runtime: 12.241723
LewlSauce
  • 5,326
  • 8
  • 44
  • 91

5 Answers5

7

I'm having exactly the same issue with my VMWare virtual machine (VMWare running on Windows).

The only workaround that is working for me is:

docker stop $(docker ps -aq) && sudo systemctl restart NetworkManager docker

If I had to guess I would say it may be related to some firewall rules docker setup on start, maybe when you resume the virtual machine a change in the network configuration breaks those rules.

Similar issue: https://github.com/docker/for-mac/issues/1990 (Doesn't seem specific to docker for mac).

jjacobi
  • 385
  • 2
  • 9
  • 1
    Works for me, Ubuntu 20.20 virtualized from W10 host with VMWare Workstation. – s3m3n Jan 13 '21 at 08:30
  • 1
    This gives additional background: https://github.com/docker/compose/issues/7405, at the end of the issue you can find a link to this gist: https://gist.github.com/lanrat/458066dbdeb460b9cef40dc2af639a24, which discusses ways to configure NetworkManager to ignore the docker interfaces. – lanoxx Feb 14 '22 at 11:13
4

I was able to solve this issue with the hint given by lannox in the comment. It's necessary to mark the network interfaces of the docker containers as unmanaged by NetworkManager.

To do that, create a new file /etc/NetworkManager/conf.d/10-unmanage-docker-interfaces.conf with the following content:

[keyfile]
unmanaged-devices=interface-name:docker*;interface-name:veth*;interface-name:br-*;interface-name:vmnet*;interface-name:vboxnet*

This configures NetworkManager to ignore all interfaces with names docker*, veth*, br-*, vmnet*, and vboxnet* interfaces.

Then restart NetworkManager with sudo systemctl restart NetworkManager.

Next time the host suspends and resumes, the docker containers keep their network connectivity.

Benedikt Köppel
  • 4,853
  • 4
  • 32
  • 42
  • 1
    This answer was immensely helpful to our organization! I work for a group that creates IT/CyberSecurity training classes. We provide a multitude of VMs to our students. Many of these run docker images for ELK/OpenSearch and tools built on this and similar platforms. This fix is super simple while being effective. THANK. YOU. !!! – user1435743 Jun 25 '22 at 05:13
0

Several questions here that might help you solve this :

  • Is your docker container still running? Run docker ps and find your container
  • Since the -p 3000:3000 option is set I guess the port is exposed, but you might want to check you really have run your container with this option this time
  • Is your app really listening? Run lsof -np | grep listen and find your app listening on port 3000
  • Connect to your container with docker exec -it <your_container> bash and try running lsof -np | grep listen to see if this is a docker issue or your app
Baptiste Merliot
  • 841
  • 11
  • 24
  • Docker container is still running (I am in a tmux session managing the application). I've also tried to shut down the rails server and re-start it. I can also confirm it's listening with `curl http://localhost:3000` and see it within the docker container. – LewlSauce Sep 10 '19 at 16:01
  • I'm not sure I can help you with the informations I have. Can you update the question with steps to reproduce? – Baptiste Merliot Sep 10 '19 at 18:51
  • 1
    Just updated the question to contain some additional information. – LewlSauce Sep 11 '19 at 15:37
0

It seems that when you run netstat on your VM you get the following line :

tcp6       0      0 :::3000                 :::*                    LISTEN      32731/docker-proxy

On Docker you get :

tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      765/puma 3.12.1 (tc

There are two differences here:

  • :::3000 vs 0 0.0.0.0:3000, the first means it is listening on IPv6 and the second on IPv4 (found the info on this question).
  • tcp6 vs tcp, again IPv6 vs IPv4.

According to this other question, it seems you have to run rails with -b :: option.

The -b option binds Rails to the specified IP, by default it is localhost. You can run a server as a daemon by passing a -d option.

Baptiste Merliot
  • 841
  • 11
  • 24
  • Not sure why it only occurs after you resume the VM though. You may want to run `netstat` before you resume just to see if it is also IPv6 vs IPv4. – Baptiste Merliot Sep 11 '19 at 16:10
  • Yeah it's very strange. I just restarted docker service and apparently it's working again. However, my netstat outputs look exactly the same. I think this may just have something to do with the way it's passed from the docker container or something. Very strange. – LewlSauce Sep 11 '19 at 16:16
  • Might be worth making a ticket on the docker github with a dummy rails app to reproduce. – Baptiste Merliot Sep 11 '19 at 16:20
  • 1
    Absolutely. I really appreciate your help! – LewlSauce Sep 11 '19 at 16:21
0

Please do

sudo docker ps

If you do not got your container do 

sudo docker ps -a

Does your container is stopped? If its true so do 

sudo docker start CONTAINER_ID
Serhii Trykoza
  • 1
  • 2