Zmq_Push and Zmq_Pull sockets

Question

I am working on a project where in I am using single zmq_push socket in multiple threads where each thread continuously send packets. On the other side I have single Zmq_Pull socket which keeps on receiving....I want this process to be faster, my application that is sending the packets thru zmq push is fast, but I doubt that my reception is faster or not....And one more thing each thread sending data is unique one, based in each thread data, I process in different thread on receiver side....how to make this process faster. I believe instead of looping on each packet on recv end I need this to done separately for each thread...Any answers on how to do this?

Answer 1

To give your receiver side more throughput add multiple PULL sockets each with its own thread or own process

Then create a zmq_proxy process with PULL and PUSH sockets. This is very simple and just a few lines of code http://api.zeromq.org/4-3:zmq-proxy

• Senders will connect to the PULL socket of the proxy
• Receiver will connect to the PUSH socket of the proxy
• The load will be balanced in a very simple round robin scheme.

+----------+     +----------+        +----------+      +----------+
|          |     |          |        |          |      |          |
| PUSH     |     | PUSH     |        | PUSH     |      | PUSH     |
|          |     |          |        |          |      |          |
+------+---+     +---------++        +-+--------+      +--+-------+
       |                   |           |                  |
       |                   |           |                  |
       |                   |           |                  |
       |                   |           |                  |
       |                   |           |                  |
       |                   |           |                  |
       |                   v           v                  |
       |                                                  |
       |                   +------------+                 |
       |                   |            |                 |
       +-----------------> |   PULL     | <---------------+
                           |            |
                           +------------+
                           |   zmq_proxy|
                           +------------+
                           |     PUSH   |
      +--------------------+            +----------------+
      |                    +-----+------+                |
      |                          |                       |
      |                          |                       |
      |                          |                       |
      |                          |                       |
      |                          |                       |
+-----v----+               +-----v----+              +---v------+
|          |               |          |              |          |
|  PULL    |               |  PULL    |              |  PULL    |
|          |               |          |              |          |
+----------+               +----------+              +----------+

NOTE: If you want more intellegent load balancing you will need to use sockets that provide feedback like ROUTER/DEALER. http://zguide.zeromq.org/page:all#The-Load-Balancing-Pattern

Answer 2

Q: how to make this process faster?
A: We need both COMMUNICATIONs and PROCESSING faster

_{If you feel the ZeroMQ concept is new to you and might be a bit tricky to master for your future work, you may like to review the key concept features in this [ The principles of the ZeroMQ hierarchy in less than a five seconds ]}

a) one can always improve the Communications-related processing capacities with instantiating the core-engine's Context( nIoTHREADs ) having nIoTHREADs >> 1

b) one may, upon a technically verified need, make each processing-thread off-loaded by a quite efficiently designed local inter-process messaging infrastructure ( to off-load communications from a thread-specific processing ) using a protocol-stack-less inproc:// communication channel from each processing-thread to the "central"-data-pumping Communicator ( Context-instance(s) may be shared among threads, while Socket-instances may not - this is the ZeroMQ Zen-of-Zero, by design ). This way each processing-thread does not block or isolate valuable resources thus made available for higher capacities of the "central"-data-pumping Communicator and may "delegate" communications related parts of the workload to the "central"-data-pump, using the lightmost send-and-forget philosophy in the processing-threads, while allowing the Communicator to handle and manage all potential smart-features, like app-level ACK/NACK signalling for dropped messages, on-the-fly service capacity scaling and many similar features each performance tuning engineer will love to have.

c) one may, upon technically indicated need, increase the performance of the receiving side ( the PULL-side ) by using a pool of PULL-receivers and make the PUSH-er side to simply .connect() to each of 'em. The internal mechanics of the PUSH-side Context-instance will need well oiled ( well parametrised ) configuration for multiple outgoing IO-channels, yet, if the remote-side PULL-ers' processing performance was the blocker, this increased and scalable pool-capacity is the way to go - sure, obviously not in the case when the PULL-side resources are already collocated on the same hardware-node with the senders, both being already at the ceiling of available processing powers ( The ZeroMQ way of thinking is always going rather distributed, than collocating processing agents on the single local hardware/OS - that is the freedom of the Zen-of-Zero evangelisation and the basis for _almost linear performance scalability ).