2

Shout out if there is something better we should consider:

I am looking for a very quick and simple way to get several programs (e.g. 5) - each running on separate nodes on a private OpenStack cloud to talk to each other.

  • Packets will be short C++ structs (less than 100 bytes)
  • Traffic will be light (probably less than 100/second)
  • Latency is really not an issue. (what is a few ms between friends?) - we have lots of cycles and memory
  • Messages should be done as pub/sub client/server paradigm
  • Library should be C++ friendly. But work both on Windows and Linux
  • We might need additional language bindings later on
  • We would prefer not to lose messages

Here is the first idea I have. But if you have something else to offer. Yell out.

Friendly Wrapper for UDP socket layer:

Encoder/Decoder for C++ struct data:

user3666197
  • 1
  • 6
  • 50
  • 92
Dr.YSG
  • 7,171
  • 22
  • 81
  • 139
  • zeromq over tcp doesn't loose messages easily, whereas nng is more likely to lose messages on a slow receiver – Thomas Jul 14 '20 at 18:04
  • Realistically, if your only sending 100 bytes per msg, 100 times per second, you can pretty much use any protocol with any form of encoding/serialization. You could do it over curl/http using JSON encoding if you wanted to. If on a private cloud 2 cores can likely handle between 50,000 to 100,000 msgs per second with a simple design, so the only thing that would make nanomsg and FlatBuffers poor candidates is their complexity when considering how simple the problem is. 100 bytes * 100 is only 10,000 bytes per sec. That wont overwhelm the IO buffers, especially not at 100 msg per second. – JSON Jul 05 '22 at 10:17
  • An 8 bit microcontroller can likely handle the load your suggesting. Your likely over complicating the problem. Note I say this because you say latency is not an issue. If latency isn't important I'm assuming that msgs timing doesn't have to be synchronized. That means 100 msgs per second can easily be done via TCP. UDP generally isn't a good candidate if loosing messages is more of an issue than latency. – JSON Jul 05 '22 at 10:33

2 Answers2

3

For serialisation, almost anything with the right language bindings will do. Google Protocol Buffers are language-agnostic, lots of bindings available. The only thing to avoid is serialisation that is built into your source code (like Boost's serialisation is / was), because then you can't readily port that to another language.

For message transport, ZeroMQ, NanoMsg are good choices. However, I think it really comes down to

  1. How badly you don't want to lose messages,
  2. Exactly what you mean by "lost message" in the first place.

The thing about ZeroMQ (and NanoMsg) is (AFAIK) there is no real way of knowing the fate of a message when a fault occurs. For instance, in ZeroMQ, if you send a message and the recipient just happens to be working and connected, the message gets transferred over the connection. The sending end now thinks that the job is done, the message has been delivered. However, unless and until the receiving end actually calls zmq_recv() and fully processes what it gets given, the message can still get lost if the receiving end process crashes, of there is a power failure, etc. This is because until it is consumed the message is stored in RAM inside the ZeroMQ run thread ( inside the respective Context()-instance's domain of control ).

You can account for this by having some sort of ack message heading back the other way, timeouts, etc. But that starts getting fiddly, and you'd be better off with something like RabbitMQ.

user3666197
  • 1
  • 6
  • 50
  • 92
bazza
  • 7,580
  • 15
  • 22
2

With an utmost respect to Margaret HAMILTON's work for Apollo Programme. enter image description here

Yell out.

Dear Dr.,
there are many features, that will become important for a professional decision, that were not mentioned in the shopping-list above.

If the Apollo AGC example could help here, the better.

The package under seek :

  • ought keep your code in control of relative processing-priorities,
  • ought let one increase / decrease / map IO-threads resources-pool usage, per channel,
  • ought let one setup the L3+ transport ToS-prioritisation tagging,
  • ought let one modify O/S configured L3 / L2 stack implementation resources,
  • ought permit non-blocking controls from application programme side, without risks of any deadlocks,
  • ought provide wide portfolio of transport-classes covering { inproc: | ipc: | tipc: | tcp: | udp: | vmci:| pgm: | epgm: } as needed to get employed in a mix

if all that sounds important for your further efforts in Draper Lab, the ZeroMQ will smoothly fit in ( if some latency / performance figures might need boosted on steroids as being more important than the mature mix above, may also enjoy to review Martin Sustrik's another kid, younger than his ZeroMQ - the tools ).

Anyway, if you indeed have (cit.) "... lots of cycles and memory ...", just let me in, I have lots of TFLOP-s to process, if you permit, in the spare time :o)

Nota Bene:

Given the recent comments, let me add a remark on using Naval Research Laboratory published NACK-Oriented Reliable Multicast (NORM) norm:// transport-class, that may help your pure PUB/SUB-design intentions, for which it seems it can fix the desire to "prefer not to lose messages" that is otherwise not taken care off, according to the Zen-of-Zero, which leaves all such operations onto the user-side application code and behaviour.

user3666197
  • 1
  • 6
  • 50
  • 92
  • Well, the humor is ok, but almost no one knows of her here. Before my time. Anyway, read the message carefully. It should be clear we have a very light load. I am now looking at NanoMSG (the next generation). – Dr.YSG Jun 17 '18 at 21:12
  • Anyway my beginnings were with DEC and Gordon Bell, if you want to go back along memory lane. – Dr.YSG Jun 17 '18 at 21:18
  • Well, Ms. Hamilton has sketched solutions to lessons, that ought be taught at the very beginning of programming. BTW, was not joking and did read well about the light workloads. Guess Martin Sústrik's younger nanomsg will be a better fit, yet the project was handed over to someone else and not so visibly active compared to the ZeroMQ momentum. Still might be fine. Have "seen" **Digital** CEO Gordon Bell recently in Ike Nassi's launch of TidalScale - was attractive for me for pico-layered inverse-hypervisor consolidation of vast-scale clusters, with smart memory-paging ( wandering CPU-state ). – user3666197 Jun 17 '18 at 21:33
  • @Dr.YSG you may have noticed, that StackOverflow-ers do not welcome Questions about some package recommendations or relative product comparisons. Take it as a sort of keen-policy-enforcers ( maybe, your [Dr.XYZ] title has prevented most of the straight haters that other suffer from for ages ) **Nota Bene :** as you posted on the wish-list, neither the ZeroMQ, nor the nanomsg / NNG will solve your "*prefer not to lose messages*" - Martin's philosophy was always the Zen-of-Zero - avoid undesired robustness for the sake of latency + performance. Add behaviour on the app-side, if / where desirable – user3666197 Jun 17 '18 at 21:42
  • Ad-hoc PUB/SUB implementation: also might have noticed, that on large scales, nanomsg has better subscription-management performance scaling due to other TOPIC-list filtering. Scaling is a deciding factor for the contemporary C/S. More recent ZeroMQ versions have re-centralised the TOPIC-list filtering back on the PUB-side ( at a cost of all centralised processing power thereof ) and reduced the original versions associated netowrk flows everything-sent-to-everybody ( at a cost of increasing network traffice niveau ) with a remote TOPIC-list filtering. May **test `norm://` transport-class**. – user3666197 Jun 17 '18 at 22:03