Bidirectional Messaging Queue

Question

This seems like a question belonging to a FAQ, but I cannot phrase the question to find it answered.
Anyway, I am currently investigating message queues (glossed over ZeroMQ, RabbitMQ tutorials) and I think MQs fit nicely into transformational dataflow - listen on Q1 for messages in format F1, transform into F2 and put transformed message on Q2. Natural question arises of what happens if transformation is inherently bidirectional and there is no well defined consumer and producer, e.g. yaml<->json converter?

As far as I understand message queues are inherently unidirectional and for bidirectional messaging I see two "solutions":

• Have two distinct input output queues, which translates into 4 in this example: json.in, json.out, yaml.in, yaml.out;
• Connect converter to both ends of JSON queue, to distinguish between raw json and converted from yaml, wrap messages in messages specifying whether this is incoming or outgoing message. But as a consequence, converter now gets a ton of messages that it has to parse and reinsert back to queue - sounds hellishly inefficient.

Former solution sounds like a way to go, unless latter is delegated to MQ broker in some form (e.g. consumer/producer IDs: json.out, id = connect_produce("json", null); json.in = connect_consume("json", id);) and broker just figures out not to send message to consumer if the message was produced by the same entity. As fas as I understand any kind of filtering will boil down to message tagging (topic exchange in RabbitMQ), which requires multiple queues.

So my question is how is this done in practice? Sticking to multiple queues or some frameworks implement bidirectional queues? Or maybe I'm missing something obvious here?

Answer 1

Using queues for translating message formats seems like overkill. That by itself sounds hellishly inefficient, so if you choose to do message translation that way, you're already down the wrong path and can choose any option you like, as they're all bad.

In terms of picking a queue design, independently of what you're doing with it, logically a bidirectional queue is the same as two one-way queues. Just go with whichever is natural for the queue implementation that you use.

Answer 2

Similar words, but different meanings.

While many pieces of software uses words like "queue", "socket", "MQ" the same spelling does not warrant the identical meaning.

Concept matters.

As a big difference between the named two, ZeroMQ is a broker-less.

The next big issue is to understand the global map of the landscape, covered with the relevant technology's concept.

For the sake of your transformation-motivated question, a simple answer would be "there is no universal magic to transform anything-in to something-out". There is always a bigger picture and a wider operational context, into which your solution has to fit in.

def mainloop():
    while True:
          ...
          if aRealTimeSCHEDULER.permitsXFORM1to2:
             try:
                 aMessage1 = Q1.recv( NOBLOCK )
                 if aMessage1 != EMPTY:
                    try:
                        aMessage2 = transform( aMessage1 )
                        Q2.send( aMessage2 )
                    except:
                        ...
                    finally:
                        ...
             except:
                 ...
             finally:
                 ...

However, in case you need not persistence/logging-broker ( which is on the other hand both a risk + performance singularity ) the ZeroMQ / nanomsg et al provide you with powerful archetypes to design ad-hoc everything needed for smart, agent-based, scale-able formal communication patterns re-used, that would allow you rapid prototype and implement your domain-specific transformation utility ( be it unidirectional, bi-directional, transactional in any sense of its meaning ).

Creating your own, domain-specific, { inproc:// | ipc:// | tcp:// | pgm:// | ... } transport-agnostic, { localhost | grid | cloud | GPGPU | FPGA } heterogeneously distribute-able, performance-wise scale-able { round-robin | weighted load-balancing }, self-healing reinstantiation / SPOF-protected, including any additional control- / self-diagnostics- / monitoring- plane(s), is adding you great powers for any transformation scheme you need to deploy in production.

---

Glue logic between json.in, json.out, yaml.in, yaml.out?

Just one picture, Fig.60 from the below-mentioned book:

---

The best next step?

To see a bigger picture on this subject with more arguments, a simple signalling-plane picture and a direct link to a must-read book from Pieter HINTJENS.

If interested in a younger sister of ZeroMQ, a nanomsg, check even a more lightweight framework from Martin SUSTRIK nanomsg.org.