How to get ZeroMQ to .connect() to localhost using a particular source IP?

Question

I have a Linux machine configured with additional IPs on the loopback interface:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 100.100.100.100/24 scope global lo
       valid_lft forever preferred_lft forever
    inet 100.100.100.101/24 scope global secondary lo
       valid_lft forever preferred_lft forever
    ...

I'm using ZeroMQ's PUB/SUB pattern to connect to remotes using each of these source ips.

socket = zmq.Context().socket(zmq.SUB)
socket.connect('tcp://100.100.100.100:5555;192.168.1.1:5555')

The corresponding server listens on all interfaces.

socket = zmq.Context().socket(zmq.PUB)
socket.bind('tcp://*:5555')

Sending using these addresses works for remote hosts given a properly-configured network, so I know this works. (It's complicated.) Now I want to unit-test this setup, which means checking that setting the source ip works without needing a remote host for the server. I run the server in the same configuration and then try to connect using something like:

socket = zmq.Context().socket(zmq.SUB)
socket.connect('tcp://100.100.100.100:5555;localhost:5555')

The client never makes the connection with the server, but if I remove the source endpoint, it works. Neither localhost, nor 127.0.0.1 work as the destination address in the .connect() call. However, if I call netcat, using the same source IP,

nc -s 100.100.100.100 -v -z -w 5 localhost 5555

This succeeds, and the server I connected to properly receives the connection as coming from 100.100.100.100. I looked at tshark's output and in the ZeroMQ client case, I don't seen any traffic from 100.100.100.100 over the loopback interface, while when I use nc to establish a TCP connection to the ZeroMQ server, I do.

What's going on here? Does ZeroMQ do something special for this kind of hairpin connection, and if so, is there a way to disable it? Is there a good way to test that I'm invoking the source IP functionality of ZeroMQ correctly without using a remote host?

This may be viewed as a follow-up to my previous question.

score 0 · Accepted Answer · answered Nov 29 '17 at 21:37

You can't send and receive on the same port on the same interface.
The best solution is to allow ZeroMQ to pick the port to send on:

socket = zmq.Context().socket(zmq.SUB) socket.connect('tcp://100.100.100.100:*;192.168.1.1:5555')

Unfortunately, the 4.2.2 release of ZeroMQ doesn't support this, though an upcoming release should. For now, the only solution is to hardcode a different port for the sending and receiving address:

socket = zmq.Context().socket(zmq.SUB) socket.connect('tcp://100.100.100.100:6666;192.168.1.1:5555')

How to get ZeroMQ to .connect() to localhost using a particular source IP?

1 Answers1