2

How does one setup multiple publishers & subscribers with a TCP transport. I suspect that you don't do automatic mesh/bus creation. So one needs a unique IP bind point for each publishers, right? They just have the subscribers connect to each publisher on a single socket.

(this was discussed in: https://www.freelists.org/post/nanomsg/does-nanomsg-support-multi-producer-in-pubsub-mode,10 )

Is that basically correct?

The reason I lean to a pub/sub over a bus/mesh approach is because (and I admit I might well be mistaken) -

  • I don't need a fully connected mesh
  • I figure your radix -tree filtering at each node is better than what I would come up with
  • I like the "auto-discovery" aspect of pub/sub as opposed to the manual wiring of the mesh for the bus transports
  • (i.e. automatic "entry and exit from mesh")

Basically what I have is 2 producers (that mainly do publishing, but can receive occasional requests to put additional info on the stream being published, so they do need to listen) And then about five consumers who mainly do receipt of the data stream from the publishers, but do need to occasional send requests to the producers.

And yes, I want the pub to be async, as as well as the recv() for the subscribe (no blocking is allowed in the context where I am using this).

So this is a bi-directional pub/sub architecture. And I am looking at the simplest way to implement this. (traffic is pretty light).

Of course a UDP transport would be nice for this, but I am not holding by breath.

Dr.YSG
  • 7,171
  • 22
  • 81
  • 139

2 Answers2

2

Easy :

Use the NN_PUB/NN_SUB as it was designed, for non-blocking, async, broadcast direction.

Use another async channel "bottom-up", be it a NN_PAIR/NN_PAIR or any another, more complicated Scalable Formal Communication Archetype pattern, like NN_PUSH/NN_PULL or a reversed NN_PUB/NN_SUB that fits both your intentions and latency goals.

Worker processes will be free to .nn_send() whenever either one feels such a need and the rest is in the hands of the "central"-process.

Given the scaling is 1:5 and workflows are light, as you have posted earlier, there are indeed no hidden traps on this, except the lost messages ( again, a principal solution for a reliable message delivery protocol was already proposed in an earlier post ).

On the "central"-node, there you just .nn_poll() these signalling channels regularly ( again, in a non-blocking async manner ) and .nn_recv() data just in a case, when a such channel has POSACK'ed a message to already be present there for being fetched and processed on your application code side.

That's all you need.

user3666197
  • 1
  • 6
  • 50
  • 92
  • I don't think NNG likes polling: https://nanomsg.github.io/nng/man/v1.0.0/nn_poll.3compat.html – Dr.YSG Jun 24 '18 at 16:31
  • @Dr.YSG NNG warning is something NNG-specific. Zero-wait polling was commonly used both in ZeroMQ and nanomsg since ever, mostly for use-cases, where low-latency code was developed. If NNG has stopped this Martin Sustrik's tradition, definitely good to know about. AFAIK the Zero-wait poll()-s were faster and cleaner to design & run, than a composition of non-blocking .recv()-s + exception handling, upon no message to read. Anyway, the core is in the use of the reverse-channel(s) design, not in the actual "mainloop"-code, there you have options as per your expressed preferences. – user3666197 Jun 24 '18 at 16:40
  • @Dr.YSG one may also notice, that using `.nn_poll()` is fully compliant with the best traditions of Martin Sustrik's design of the nanomsg engine, whereas the NNG intents ( as expressed in the linked content ) are to move users away from using nanomsg-style API/code-base in the favour of re-factoring the code-base to start using a completely different philosophy, as Garett D'Amore has crafted in his re-design into the NNG style of signalling / messaging ( that requires one to adopt new paradigms, at a cost of additional expenses to pass another learning curve of finding NNG best practices ) – user3666197 Jun 24 '18 at 16:50
  • understood, But what do you think of the multiple fan-in publishers to a single subscriber as discussed in: https://www.freelists.org/post/nanomsg/does-nanomsg-support-multi-producer-in-pubsub-mode,10 ) – Dr.YSG Jun 24 '18 at 16:52
  • also, I think you might be able to get the effect of the poll via a NNG_FLAG_NONBLOCK as in: https://nanomsg.github.io/nng/man/v1.0.0/nng_recv.3.html but he does not tell you what error code is returned for no data present. – Dr.YSG Jun 24 '18 at 16:54
  • @Dr.YSG cannot serve you Sir in this. The freelists pages get rejected here for security reasons. As per NNG documentation, have loaded a nicely formatted book, as was published, yet have there found typos / errors and did not have time to re-validate the actual behaviours, the less to sponsor QA / TQM experiments or tweak performance envelopes. Sure, if this was profesionally contracted, the outcomes may provide insights to the list of best practices - design-wise, performance-wise, latency-wise. – user3666197 Jun 24 '18 at 17:00
  • On Dec 29, 2015, at 11:02 PM, Dan Liu <139250065@xxxxxx> wrote: Do you mean a single consumer socket can connect different producer URLs at same time ? ------------------ Original ------------------ From: "Garrett D'Amore";; Date: Wed, Dec 30, 2015 02:49 PM To: "nanomsg"; Subject: [nanomsg] Re: does nanomsg support multi producer in pub-sub mode? If you have multiple producers the they will all need to call bind (with different URLs). You can connect a single socket to multiple URLs. – Dr.YSG Jun 24 '18 at 17:05
  • @Dr.YSG The "reversed" (**many**-worker-side-) **`NN_PUB`**-s : (**1**-central-) **`NN_SUB`** was proposed as a preferred choice in the answer above ( without a chance to read the freelists.org texts, as this is a common use-case since ever in ZeroMQ ). Given the `NN_PUB`-s do `.connect()` and the `NN_SUB` does `.bind()`, the scenario is clean and sound ( if the NNG's **`nng_dial()`** + **`nng_listen()`** methods cannot serve this way, the point looses one degree of freedom in the architecture design ). **Typically the point is who knows the counterparty's AccessPoint's address and who not.** – user3666197 Jun 24 '18 at 17:12
  • If asking about one AccessPoint's multiple calls to `.connect()`-s ( `nng_dial()`-s in NNG dialect ), yes, this was an elementary practice since ZeroMQ API v2.+ – user3666197 Jun 24 '18 at 17:19
0

NanoMSG NNG is very symmetric and orthogonal. I had a couple discussions with gdamore.

Each Endpoint can be push, req, bus, or whatever You can have multiple endpoints (e.g. 2 push, or a push and req/res) Protocol can be in either direction Rec can be blocking, async, or poll based.

If you want to see how to do this, see:

https://github.com/nanomsg/nng/issues/551

Dr.YSG
  • 7,171
  • 22
  • 81
  • 139