Can .select() on both traditional and ZeroMQ sockets?

Question

I have a thread that waits for signaling input via ZMQ and Network IO via TCP ( and another doing the same with UDP ).

socket_tcp = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
socket_tcp.connect( ( self.config.tcp_ip, self.config.tcp_port ) )

while True:
    s_ready = zmq.select( [socket_zmq, socket_tcp], [], [] )[0]
        for sock in s_ready:
            # do stuff

However, the TCP socket is never returned as writable ( I made sure it actually gets data using Wireshark ).

The Documentation says I could pass

A zmq.Socket or any Python object having a fileno()
method that returns a valid file descriptor.

and I guess the later is true for socket.socket.

Its the same with UDP.

Am I missing something?

Do I need to handle the ZMQ socket in a separate thread and pass its messages over using ctrl_rcv, ctrl_snd = multiprocessing.Pipe()?

Or can I use a select over both worlds, as I would expect?

score 2 · Accepted Answer · answered Jan 01 '18 at 23:04

After playing around with asyncio I discovered that zmq.asyncio works perfectly fine with my use case. It makes use of zmq.Poller() (as does zmq.select). But for some reason it just works...

So I ended up using ZMQSelector() with a few additional lines of code:

import selectors
from zmq.asyncio import ZMQSelector

ctx = zmq.Context()
mixed_selector = ZMQSelector()
mixed_selector.register(sock_tcp, selectors.EVENT_READ)
mixed_selector.register(sock_zmq, selectors.EVENT_READ)

while True:
    fd_event_list = mixed_selector.select()
    for key, event in fd_event_list:
        sock = key.fileobj
        # do stuff

score -1 · Answer 2 · edited Jan 08 '18 at 22:46

-1

Do not panic ( yet )

Well, while the recent ZeroMQ API does offer an ZMQ_FD-file-descriptor proxy for touching the ZeroMQ socket instances, these instances are just a tip of a "hidden"-iceberg and behave a way different from any other "common"-socket abstraction, as published here, so one ought better straight forget ZeroMQ to be "something-like-socket"

Best
be warned about at least
the main conceptual differences in the [ ZeroMQ hierarchy in less than a five seconds ] Section.

The powers of the ZeroMQ framework tools are in their concise design, so it is almost sure, it will lose all these unique advantages, if used just "side-by-side" ( in whatever geeky syntax-constructor trick ) with any principally way-simpler ( primitive, if one wishes so ) signalling / messaging device.

If one took all efforts so far to implement a ZeroMQ signalling infrastructure, it would be a nonsense not to use it.

So rather deliver inter-thread messaging again better over ZeroMQ, best using zero-copy smart inproc:// transport-class and avoid any platform-specific overheads a multiprocessing.Pipe() will additionally create just for moving ( ... and replicating ) the already known pieces of data ( ZeroMQ may enjoy an ultra-low latency right due to zero-copy, where applicable ).

edited Jan 08 '18 at 22:46

halfer

19,824
17
99
186

answered Dec 30 '17 at 18:56

user3666197

1
6
50
92

Well yes, that is the reason I use ZeroMQ. – Drak Dec 30 '17 at 23:34
Glad it helped. – user3666197 Dec 30 '17 at 23:37
[Hit enter to early] But at one point in the application it has to meet classic sockets (in an IO thread). And I could use a ``select.select`` on a Pipe but not on ZMQ. So the question is: What is the best way to get the signaling (originating from ZMQ) in the IO thread (handling the normal socket)? – Drak Dec 30 '17 at 23:42
Do not hesistate to check other my posts on **non-blocking schedulers**, where ZeroMQ socket-IO-s take place upon a just `.poll()`-signalled arrival of anything useful to take care of, never else, and feel free to read also about other high performance tricks. Btw, Draku, **what are your expected throughput `[GB/s]` and latency `[us]` IO-thread performance envelopes?** – user3666197 Dec 30 '17 at 23:47

Can .select() on both traditional and ZeroMQ sockets?

2 Answers2

Do not panic ( yet )