Docker overlay networking centos7

Question

After pulling out all my hair. Trying to get docker overlay network on a 3 node CentOS7 swarm cluster, I need some help please.

1 - Upgraded kernel on all nodes 4.5.0-1.el7.elrepo.x86_64
2 - Docker Version : Docker version 1.10.3, build 20f81dd (same on all nodes) Docker info:

Filters: health, port, dependency, affinity, constraint<br>
Nodes: 3<br>
 docker1: 192.168.1.231:2375<br>
  └ Status: Healthy<br>
  └ Containers: 5<br>
  └ Reserved CPUs: 0 / 1<br>
  └ Reserved Memory: 0 B / 2.033 GiB<br>
  └ Labels: executiondriver=native-0.2, kernelversion=4.5.0-1.el7.elrepo.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper<br>
  └ Error: (none)<br>
  └ UpdatedAt: 2016-04-04T03:16:04Z<br>
 docker2: 192.168.1.35:2375<br>
  └ Status: Healthy<br>
  └ Containers: 3<br>
  └ Reserved CPUs: 0 / 1<br>
  └ Reserved Memory: 0 B / 2.033 GiB<br>
  └ Labels: executiondriver=native-0.2, kernelversion=4.5.0-1.el7.elrepo.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper<br>
  └ Error: (none)<br>
  └ UpdatedAt: 2016-04-04T03:16:12Z<br>
 docker3: 192.168.1.36:2375<br>
  └ Status: Healthy<br>
  └ Containers: 2<br>
  └ Reserved CPUs: 0 / 1<br>
  └ Reserved Memory: 0 B / 2.033 GiB<br>
  └ Labels: executiondriver=native-0.2, kernelversion=4.5.0-1.el7.elrepo.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper<br>
  └ Error: (none)<br>
  └ UpdatedAt: 2016-04-04T03:16:20Z<br>
Plugins:<br>
 Volume:<br>
 Network:<br>
Kernel Version: 4.5.0-1.el7.elrepo.x86_64<br>
Operating System: linux<br>
Architecture: amd64<br>
CPUs: 3<br>
Total Memory: 6.099 GiB<br>
Name: 21131682ca99<br>

Creating the overlay network works fine

docker network create -d overlay --subnet=10.10.10.0/24 RED

Then create two test containers, one on each docker node:

docker1
 docker run -itd --name container1 --net RED busybox<br>
docker2
 docker run -itd --name container2 --net RED busybox<br>

Check network:

docker -H :4000 network inspect RED
[
    {
        "Name": "RED",
        "Id": <br>"f64fcb769bba4a4b0d709599e9670657ee2cba3c98ca031909bc3967b6085867",
        "Scope": "global",
        "Driver": "overlay",
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.10.10.0/24"
                }
            ]
        },
        "Containers": {
            "1371c2fec957fe2ecd9d9d9dcf21492e63ef497c662a4b7eee7e8170738875b2": {
                "Name": "container1",
                "EndpointID": "e1e5d0d0c794da9f7a9054e0012bb184b10501c4cdd537193608a38bafa1ee00",
                "MacAddress": "02:42:0a:0a:0a:02",
                "IPv4Address": "10.10.10.2/24",
                "IPv6Address": ""
            }
  "489c7963b7484041a7ef7647b7383a69dfdde8eb416e0a26aa41a023e154ab5f": {
                "Name": "container2",
                "EndpointID": "3d3afb54922d4b904056fc637e7f76cbfe632230a730860ecde88e12ec9f16f2",
                "MacAddress": "02:42:0a:0a:0a:03",
                "IPv4Address": "10.10.10.3/24",
                "IPv6Address": ""

I'm unable to ping from one container to another across the different hosts. It does of course work when both containers are on the same host. Any help/suggestions please.

Answer 1

Don't forget that a multi-machine overlay network supposes you need a key-value store, as I showed in "How to make Docker container accessible to other network machines through IP?" and in this tutorial.

That means your docker daemon must be started with a reference to that key-value store.

See also "Swarm and container networks":

Multi-host networks require a key-value store.
The key-value store holds information about the network state which includes discovery, networks, endpoints, IP addresses, and more.

You configure the Docker Engine daemon to use this store.
Two required parameters, --cluster-store and --cluster-advertise, refer to your key-value store server.