176

Containers in a host "suddenly" loses connection to outside-world containers. However, some hosts were refreshed and suddenly we had the following situation:

  1. The host can communicate with other hosts.
  2. Containers running in the host cannot communicate with other hosts.

Here's an example:

[root@pprdespap322 deploy]# ping ci.docker.company.net
PING pprdespap324.corp.company.net (10.137.55.22) 56(84) bytes of data.
64 bytes from pprdespap324.corp.company.net (10.137.55.22): icmp_seq=1 ttl=64 time=0.282 ms
64 bytes from pprdespap324.corp.company.net (10.137.55.22): icmp_seq=2 ttl=64 time=0.341 ms
^C
--- pprdespap324.corp.company.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.282/0.311/0.341/0.034 ms

Now, from the container itself, we cannot ping the same host:

[root@pprdespap322 deploy]# docker run -ti quay.io/coreos/registry ping ci.docker.company.net
WARNING: IPv4 forwarding is disabled. Networking will not work.
ping: unknown host ci.docker.company.net

The first time I saw this warning was in the initial versions of Docker... Having Docker 1.9.1 and 1.10.3, How to solve this problem?

Marcello DeSales
  • 21,361
  • 14
  • 77
  • 80
  • Note: I get this problem intermittently when connecting / disconnecting from an AWS vpn for work. It's [documented](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/what-is.html#what-is-limitations) that IP forwarding is disabled by the AWS VPN Client. – Jeff Ward Apr 11 '23 at 13:15

5 Answers5

252

I reviewed http://chrisgilmerproj.github.io/ubuntu/network/docker/2013/09/05/ipv4-forwarding-and-docker.html and it helped me solving the problem on the host.

I added the following to /etc/sysctl.conf: 

net.ipv4.ip_forward=1

I then restarted the network service and validated the setting:

[root@pprdespap322 deploy]#  systemctl restart network
[root@pprdespap322 deploy]# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
[root@pprdespap322 deploy]# docker run -ti quay.io/coreos/registry ping ci.docker.company.net
PING pprdespap324.corp.company.net (10.137.55.22) 56(84) bytes of data.
64 bytes from pprdespap324.corp.company.net (10.137.55.22): icmp_seq=1 ttl=63 time=0.329 ms
64 bytes from pprdespap324.corp.company.net (10.137.55.22): icmp_seq=2 ttl=63 time=0.306 ms
^C
--- pprdespap324.corp.company.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.306/0.317/0.329/0.021 ms

All containers now can communicate with outside world containers!

Marcello DeSales
  • 21,361
  • 14
  • 77
  • 80
  • 30
    On Ubuntu I needed `sudo sysctl -p` to reload the settings (systemctl was not found). – Nacho Coloma Apr 22 '17 at 12:40
  • 2
    This solution was needed on Centos 7 when simply trying to access the container from the host (for testing). – Dave C May 22 '17 at 14:46
  • I'm having similar in containers running on a Windows host (on Hyper-V). Is there a similar setting somewhere for Windows? – Anthony Mastrean Sep 07 '17 at 15:58
  • in centos 7: net.ipv4.ip_forward = 1 into file /usr/lib/sysctl.d/50-default.conf – Lei Yang Nov 08 '18 at 07:48
  • 2
    @user3338098, because the underlying internal network is forwarding traffic between interfaces to gain internet access. This has been needed to indicate to any linux system that it is to route traffic between interfaces in the way that a network router does for a very long time. This is turned off by default because most linux boxes do no forwarding and to forward traffic accidentally would be a security threat at worst or really complicate your network at best. – Josiah Jul 23 '19 at 00:14
  • 1
    I saw a cool option [here](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section) `[runners.docker.sysctls] "net.ipv4.ip_forward" = "1"`, but for some reason didn't work for me. Is there any other way to let container use network not changing the global host settings and when container is run not from command line, but from `gitlab-runner`, so that I can't give it `--network` parameter? – RAM237 Oct 10 '19 at 23:33
  • FYI, Instead of `systemctl restart network` I had to run `systemctl restart networking` in ubuntu 18.04 – Ganesh Satpute Sep 29 '20 at 08:46
110

Try restarting Docker service.

E.g. for Ubuntu:

$ sudo systemctl restart docker
Phoenix
  • 3,996
  • 4
  • 29
  • 40
DmitrySandalov
  • 3,879
  • 3
  • 23
  • 17
  • 12
    On CentOS7 this problem suddenly appeared and simply restarting the docker service worked perfectly. – steven87vt Dec 04 '19 at 13:54
  • 5
    This worked, makes me nervous that I don't know why it worked. Anyone have a clue? Or at least steps to repeat the failure? – Josiah Jan 17 '20 at 21:21
  • 1
    reproduction? It happened here after upgrading Docker and rebooting (on Oracle Linux Server release 7.8 with kernel: - 3.10.0-1127.el7.x86_64 #1 SMP Wed Apr 1 10:20:09 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux - upgrade Docker to: docker-ce-3:19.03.8-3.el7.x86_64 via yum repo: @docker-ol7-prod ) – JohannesB Jul 23 '20 at 11:28
  • 3
    It works, thanks! But anyone knows how it did the job? – c0degeas Aug 18 '20 at 10:43
  • 1
    Worked to me too, any idea why is happens? – assli100 Nov 16 '21 at 14:24
  • 3
    Only now, after MANY HOURS spent on issues today, I realized that docker runs `sysctl -w net.ipv4.ip_forward=1` when the Daemon starts up, while I have `net.ipv4.ip_forward=0` in my sysctl.conf file. This explains why I get sudden problems with containers not accessing the outside network, "randomly"... Finally a long-term mystery resolved! – Nuno Aug 07 '22 at 02:14
  • This definitely helps, but does anybody have any clues why situation repeats (for every few days)? I have `net.ipv4.ip_forward = 1` in the host system, but IPv4 forwarding just disables on it's own until restart – ololobus Nov 18 '22 at 19:57
28

Try adding --network=host along with docker run command to fix this.

https://medium.com/@gchandra/docker-ipv4-forwarding-is-disabled-8499ce59231e

Veda
  • 2,025
  • 1
  • 18
  • 34
Ganesh Chandrasekaran
  • 1,578
  • 12
  • 17
  • 2
    Useful for a single container on run. The parameter is `--net=host` – Max13 Mar 29 '19 at 10:18
  • 1
    Now problem with this approach is if you have multiple containers running in parallel, like on Jenkins, ports are shared and you risk hitting issues on those shared ports. To be specific we are running e2e tests with Cypress and we are getting issues with Xvfb not able to spawn new instance as ports are already taken. – Alex Rashkov Mar 19 '20 at 12:44
  • This helped me resolve an error with Maven and fabric8io where installing software while building the image failed because the rpm repository could not be found. – Markus Rohlof Jul 19 '21 at 14:27
  • 2
    works on docker build too, thanks! – Yablargo Sep 30 '21 at 17:24
  • This worked great with the docker-in-docker (dind) configuration in Gitlab for a docker build. Thank you! – Khoward Oct 21 '21 at 21:48
5

Solved my issue restarting the network.

systemctl restart network
Cristiano Kubiaki
  • 51
  • 1
  • 1
2

What worked for me:

  1. Edit /etc/sysctl.conf and ensure the following line is present / uncommented:
net.ipv4.ip_forward=1

  1. Run sudo sysctl -p so changes take effect.

  2. Restart the affected container:

docker restart [container_name/id]
Jeff Ward
  • 16,563
  • 6
  • 48
  • 57
Nada Magdy
  • 21
  • 2