1

I am building a simple star-like client-server topology.

The idea is that clients connect to the server, can send messages, and the server can send messages to them, when the server decides to. There will be a relatively small number of clients, about 30, but so many that it is not sensible to send all outgoing data to all. I'm sure I'm just boneheaded, but this seems to be completely impossible with ZeroMQ.

The last part is the reason this question does not provide answer.

The catch is this :

I can use a ROUTER socket to receive messages from clients. This also carries identification. However, I cannot use the same socket for sending, since ZeroMQ sockets are not threadsafe. I.e. I can't have one thread waiting for incoming messages, and another sending outgoing from the server itself. I am not aware of any way I could wait in blocking for both - socket.recv(), and for example .get() on a queue - at the same time on a single thread in python. Maybe there is a way to do that.

Using two sockets - one incoming one outgoing - doesn't work either. The identification is not shared between sockets, and so the sending socket would still have to be polled to obtain client id mapping, if even for once. We obviously can't use own port for each client. There seems to be no way for the server to send a message to a single client out of it's own volition.

(subscription topics are a dead idea too: message filtering is performed on client-side, and the server would just flood all client networks)

In the end TCP sockets can handle this sort of asynchronous situation easily, but effective message framing on python is a nightmare to build. All I'm essentially after is a reliable socket that handles messages, and has well defined failure modes.

user3666197
  • 1
  • 6
  • 50
  • 92
Elmore
  • 256
  • 3
  • 7
  • hmm. It seems that one *could* use one router socket against outside world call this S, and work like this: Create one auxiliary socket pair for process internal communication. Call this A-B, put outgoing messages into A, then poll on a union of sockets (S,B), if S, receive that from outside, or if B, take that and put it into S. Now only one thread ever touches S, or any other socket, and we have knowledge of client identifiers. – Elmore Dec 15 '20 at 06:37

2 Answers2

1

I don't know Python but for C/C++ I would use zmq_poll(). There are several options, depending on your requirements.

  • Use zmq_poll() to wait for messages from clients. If a message arrives, process it. Also use a time-out. When the time-out expires, check if you need to send messages to clients and send them.
  • zmq_poll() can also wait on general file descriptors. You can use some type of file descriptor and trigger it (write to it) from another process or thread when you have a message to send to a client. If this file descriptor is triggered, send messages to clients.
  • Use ZeroMQ sockets internally inside your server. Use zmq_poll() to wait both on messages from clients and internal processes or threads. If the internal sockets are triggered, send messages to clients.

You can use the file descriptor or internal ZeroMQ sockets just for triggering but you can also send the message content through the file descriptor or ZeroMQ socket.

rveerd
  • 3,620
  • 1
  • 14
  • 30
  • Thank you. Good ideas, my personal ranking for them: 1) Not viable really. If latency is a thing we care about, and latency is on 10s of milliseconds, we should loop around at 100Hz just to not make most of latency come from here. 2) Good idea. Im using python though, where this is either not possible, or non-documented and hard. 3) What I ended up using. The idea to just pass triggers is a great improvement though. Thx. – Elmore Dec 16 '20 at 12:16
  • @Elmore if you think the answer is useful, to you or others, consider upvoting it. – rveerd Dec 17 '20 at 10:47
  • @Elmore: if you think your solution might be sufficiently interesting to future readers, and it differs somewhat from the existing answers, do please add your own answer if you can. Self-answered questions are most welcome here. – halfer Dec 20 '20 at 18:36
0

Q : "ZeroMQ: How to construct simple asynchronous broker?"

The concept builds on a few assumptions that are not supported or do not hold :

a)
Python threads actually never execute concurrently, they are re-[SERIAL]-ised into a sequence of soloists execution blocks & for any foreseeable future will remain such, since ever & forever (as Guido van ROSSUM has explained this feature to be a pyramidal reason for collision prevention - details on GIL-lock, serving this purpose, are countless )

b)
ZeroMQ thread-safeness has nothing to do with using a blocking-mode for operations.

c)
ZeroMQ PUB/SUB archetype does perform a topic-filtering, yet in different versions on different sides of the "ocean" :

Until v3.1, subscription mechanics ( a.k.a. a TOPIC-filter ) was handled on the SUB-side, so this part of the processing got distributed among all SUB-s ( at a cost of uniformly wide data-traffic across all transport-classes involved ) and there was no penalty, except for a sourcing such data-flow related workload ... on the PUB-side.

Since v3.1, the TOPIC-filter is processed on the PUB-side, at a cost of such a processing overhead & memory allocations, but saving all the previously wasted transport-capacities, consumed just to later realise at the SUB-side the message is not matching the TOPIC-filter and will be disposed off.

Using a .poll()-based & zmq.NOBLOCK-modes of .recv()- & .send()-methods in the code design will never leave one in ambiguous, the less in an unsalvagable deadlock waiting-state and adds the capability to design even a lightweight priority-driven soft-scheduler for doing so with different relative priority levels.

Given your strong exposure in realtime systems, you might like to have a read into this to review the ZeroMQ Framework properties.

user3666197
  • 1
  • 6
  • 50
  • 92
  • Hmm, thank you for an answer and the links provided. To rephrase my original problem: While ZeroMQ provides several different socket types, consider the set of requirements: 1) Clients may be behind NAT, so cannot host for example PULL socket themselves, 2) We want to send urgent packets to clients immediately when server knows of them, rendering req-rep initiation by client ineffective. Then how does one tackle this? I am fully aware of GIL in python, but concurrent recv- and send- calls still crash the interpreter in tests. – Elmore Dec 15 '20 at 08:19
  • 1
    Additionally, a similar question was asked here: https://stackoverflow.com/questions/59847939/how-to-create-zeromq-socket-suitable-both-for-sending-and-consuming?rq=1 And you posted there too. I'm getting the impression you know your share, but none of your replies actually answer any questions at all. A simple example here would have been sufficient. – Elmore Dec 15 '20 at 08:42