0

I tried to run docker swarm over two different networks.

  • First network is 10.10.100.x/24
  • Second network is 10.10.150.x/24

Both networks can see each other. There are no firewall rules between them to block any traffic. Specifically I tested 7946 TCP and UDP and 4789 UDP. I can confirm, that I can connect from the first network to the second network on both ports and both protocols. And also from the second network to the first network without any issue.

Docker swarm is up and running and used engine is 20.10.11 I can see that all nodes have status=Ready and availability=Active.

Ingress network is default:

enter image description here

and I can see all peers listed there as well.

But when I deploy any service to any node with port -p 20000:80, then I can see this node only from the network where it was deployed.

  • If service lands on the first network, it is accessible only through nodes from the first network, not from the second.
  • If service lands on the second network, it is accessible only through nodes from the second network, not from the first.

enter image description here

Any thoughts how to fix this?

Thanks

update 1:

Tried to run swarm with additional parameter docker swarm init --default-addr-pool 172.100.0.0/16. Result remains the same.

update 2: Based on the advice from @BMitch

I verified with sudo tcpdump -nn -s0 -v port 4789 or 7946, that port 7946 works (UDP and TCP). I also verified with the previous tcpdump command and nc -z -v -u 10.10.150.200 4789 (run from the first network), that port 4789 works as well.

suchoss
  • 3,022
  • 1
  • 19
  • 21
  • How did you connect to port 4789/udp? Udp is a connectionless protocol. – BMitch Nov 29 '21 at 18:24
  • Setup tcpdump on each side of the connection, and verify packets for the overlay ports sent from one node are received by the destination node. – BMitch Nov 29 '21 at 18:26
  • @BMitch to "connect" to UDP I used some command I found earlier. Unfortunately can't remember what was the command. I also never worked with tcpdump. How do I set it up for swarm? Should I set listening on my physical network adapter or some docker network (docker_gwbridge)? And then how do I sent those packets? Thanks – suchoss Nov 29 '21 at 19:36
  • You run it on the hosts: https://stackoverflow.com/a/60497618/596285 – BMitch Nov 29 '21 at 20:11
  • What ip addresses did you use in the docker swarm init / docker swarm join commands? wrt --advertizse-addr etc? – Chris Becke Nov 30 '21 at 04:48
  • @BMitch I tested it with `tcpdump` and it seems that ports are opened (update 2) – suchoss Nov 30 '21 at 08:46
  • @ChrisBecke The default ones. I just run `docker swarm init`, it generates join command and I use this command on worker nodes. It creates whole swarm for me. I also tried it with `--default-addr-pool 172.100.0.0/16`, but the behavior remained the same. – suchoss Nov 30 '21 at 08:48

2 Answers2

0

Same issue for me, routing and overlay work great but ingress load balancer only works through the same site endpoints that runs the container.

Oddly I discovered ingress load balancer works cross sites when using nc -l as server socket, making the whole even more obscure to me.

REM: Underlay network is wireguard VPN (L3 point-to-point)

user4374365
  • 1
  • 1
0

In the end - problem was in the NAT. Our second network was behind NAT, which caused this issue. Once we removed NAT, everything worked.

suchoss
  • 3,022
  • 1
  • 19
  • 21
  • I have the same issue, could you please advise how did you removed the NAT? is it from iptables? –  Aug 30 '22 at 17:30
  • 1
    @Igbaryya Sorry, I do not know how (it was done by my team mate). You can try to ask a new question here on the StackOverflow. – suchoss Aug 30 '22 at 18:56