How to communicate multiple images using ZeroMQ in C++?

Question

I want to implement the program that can communicate multiple images via ZeroMQ. I can display multiple images on C++ by using OpenCV but I cannot display on Python somehow although the compiler didn't output errors.

How can I communicate multiple images from C++ to Python?

C++ Code:

    Mat img0;
    Mat img1;
    Mat img2;

    img0 = imread("images.jpeg");
    img1 = imread("images1.jpeg");
    img2 = imread("image2.jpg");

    if(img0.empty()) return -1;
    if(img1.empty()) return -1;
    if(img2.empty()) return -1;
    ShowManyImages("IMAGE",3,img0,img1,img2);

// Let structure for zeroMQ
    int32_t info[3];

    info[0] = (int32_t)img0.rows;
    info[1] = (int32_t)img0.cols;
    info[2] = (int32_t)img0.type();

    info[3] = (int32_t)img1.rows;
    info[4] = (int32_t)img1.cols;
    info[5] = (int32_t)img1.type();

    info[6] = (int32_t)img2.rows;
    info[7] = (int32_t)img2.cols;
    info[8] = (int32_t)img2.type();


    zmq::context_t context (1);
    zmq::socket_t socket (context, ZMQ_REQ);
    socket.connect ("tcp://localhost:5555");

    for(i=0; i<9; i++ )
    {
        zmq::message_t msg0 ( (void*)&info[i], sizeof(int32_t), NULL  );
        socket.send(msg0, ZMQ_SNDMORE);
    }


    void* data0 = malloc(img0.total() * img0.elemSize());
    memcpy(data0, img0.data, img0.total() * img0.elemSize());

    zmq::message_t msg00(data0, img0.total() * img0.elemSize(), my_free, NULL);
    socket.send(msg00);


    void* data1 = malloc(img1.total() * img1.elemSize());
    memcpy(data1, img1.data, img1.total() * img1.elemSize());

    zmq::message_t msg11(data1, img1.total() * img1.elemSize(), my_free, NULL);
    socket.send(msg11);

    void* data2 = malloc(img2.total() * img2.elemSize());
    memcpy(data2, img2.data, img2.total() * img2.elemSize());

    zmq::message_t msg22(data2, img2.total() * img2.elemSize(), my_free, NULL);
    socket.send(msg22);

Python Code:

import zmq
import cv2
import struct
import numpy as np

# Connection String
conn_str      = "tcp://*:5555"

# Open ZMQ Connection
ctx = zmq.Context()
sock = ctx.socket(zmq.REP)
sock.bind(conn_str)


while(True):
# Receve Data from C++ Program
    byte_rows, byte_cols, byte_mat_type, data=  sock.recv_multipart()

# Convert byte to integer
    rows = struct.unpack('i', byte_rows)
    cols = struct.unpack('i', byte_cols)
    mat_type = struct.unpack('i', byte_mat_type)

    if mat_type[0] == 0:
    # Gray Scale
        image = np.frombuffer(data, dtype=np.uint8).reshape((rows[0],cols[0]))
    else:
    # BGR Color
        image = np.frombuffer(data, dtype=np.uint8).reshape((rows[0],cols[0],3))

    cv2.imshow('Python',image)
    cv2.waitKey()

Sincerely,

Would you post what exact values have you received on the python side from the posted C++ / python distributed-computing code mock-up? — user3666197, Oct 24 '19 at 04:23
I modified from `REQ` to `XREQ`.Then, I can watch three windows that displays `img0` only although. But I want to watch each windows that displays each pictures like `img0`,`img1`,`img2`. I think this is happened by `sock.recv_multipart()` because c++ program send each images information and I recovered Python program from `byte_rows, byte_cols, byte_mat_type, data= sock.recv_multipart()` to `byte_rows0, byte_cols0, byte_mat_type0, byte_rows1, byte_cols1,byte_mat_type1, byte_rows2, byte_cols2, byte_mat_type2, data= sock.recv_multipart()`. How do I fix this? — user36327, Oct 24 '19 at 06:03

user3666197 · Answer 1 · 2019-10-24T13:33:41.730

Q : How can I communicate multiple images from C++ to Python?

The code above uses REQ/REP Scalable Formal Communication Pattern Archetype, yet it does not try to send and REP-ly to any already delivered REQ-est.

Without an attempt to .send() a REP-ly, the REQ-side will never be able to send any next message.

_{In case this is your first experience of working with ZeroMQ,

you may here enjoy to first look at "ZeroMQ Principles in less than Five Seconds" before diving into further details}

Solution ?

Try to use rather a PUSH/PULL Archetype, where no response is needed.

Next, try to decide, whether it is a reasonably robust assumption to send all meta-data just with the first image ( I would rather compose, best using struct.pack()/struct.unpack() compliant BLOB with all the byte_rows, byte_cols, byte_mat_type, data in everyone, single-frame, trivial message payload mapping ).

Python side will just read a single-frame BLOB in one step, may test for a signature, if needed, .unpack() the leading few bytes to learn the image sizing, .unpack() the "rest" with the image-data and you are done. Isn't this cute?

Also be warned, that not all platforms share the same type of Endian-ness ( so best design your c-side code with explicit byte-ordering into the "network"-Endian ordering :o) just for being on the safe side if hosting devices / platforms evolve further and may become to have a different Endian-ness next time. Explicit ordering is then the same explicit ordering, as your code has insisted on using since Day 1 )

_{I see. So you say I must use PUSH/PULL right? but I modified my program from ZMQ_REQ to ZMQ_XREQ in c++ and from zmq.REP to zmq.XREQ in python. Then, I can watch three windows in python but display img0 only. I know this is happened because of sock.recv_multipart().But I don't know how I modify this. Also should I implement using PUSH/PULL? – user36327 7 hours ago}

No, I did not tell anyone there is a MUST to use PUSH/PULL, it was an advice from a guy, who uses ZeroMQ since v2.1+

XREQ/XREP have effectively short-circuited the dFSA stepping of theoretically infinite chain of REQ.send()s-REP.recv()s-REP.send()s-REQ.recv()s-REQ...-REP...-REP...-REQ...-REQ...-...-
[... ad-infinitum-or-deadlock, whichever comes first :o) ]

The problem is in expecting things to be sure to happen, which actually need not happen as one expected:

MASK        = 'Python: The Image#{0:} was decoded to be {1:}'
order_ORDER = 0
while( True ):  # THIS WILL RUN FOREVER, WON'T IT?
    #####################################################################################
    order_ORDER += 1
    #####################################################################################
    ( byte_rows,                                  # EXPECTS ALL THIS TO .recv() EACH TIME
      byte_cols,                                  # EXPECTS ALL THIS TO .recv() EACH TIME
      byte_mat_type,                              # EXPECTS ALL THIS TO .recv() EACH TIME
      data                                        # EXPECTS ALL THIS TO .recv() EACH TIME
      )           =  sock.recv_multipart()        # in Data from C++ Program
    #####################################################################################
    rows     = struct.unpack( 'i', byte_rows)     # EXPECTS THIS TO BE CORRECT  EACH TIME
    cols     = struct.unpack( 'i', byte_cols)     # EXPECTS THIS TO BE CORRECT  EACH TIME
    mat_type = struct.unpack( 'i', byte_mat_type) # EXPECTS THIS TO BE CORRECT  EACH TIME
    #####################################################################################
    if mat_type[0] == 0:                          # IF mat_type[0] WAS == 0 ?in 2nd, 3rd?
        # # # # # # # # # # # # # # # Gray Scale: # IF mat_type[0] WAS == 0 ?in 2nd, 3rd?
        image = np.frombuffer( data,
                               dtype = np.uint8
                               ).reshape( ( rows[0],
                                            cols[0]
                                            )
                                          )
        print "IMAGE WAS of TYPE == ZERO, HAVING SHAPE OF: ", image.shape, image.dtype
        imgType = 'Grayscale'
    else:                                         # IF mat_type[0] WAS != 0 ?in 2nd, 3rd?
        # # # # # # # # # # # # # # # # BGR Color # IF mat_type[0] WAS != 0 ?in 2nd, 3rd?
        image = np.frombuffer( data,
                               dtype = np.uint8
                               ).reshape( ( rows[0],
                                            cols[0],
                                            3
                                            )
                                          )
        print "IMAGE WAS of TYPE != ZERO, HAVING SHAPE OF: ", image.shape, image.dtype
        imgType = 'Blue-Green-Red'
    ###################################################################################
    cv2.imshow( MASK.format( imgType, order_ORDER ), image )
    cv2.waitKey()
    ###################################################################################
    #   LOOP AGAIN
    ###################################################################################

I see. So you say I must use `PUSH/PULL` right? but I modified my program from `ZMQ_REQ` to `ZMQ_XREQ` in c++ and from `zmq.REP` to `zmq.XREQ` in python. Then, I can watch three windows in python but display `img0` only. I know this is happened because of `sock.recv_multipart()`.But I don't know how I modify this. Also should I implement using `PUSH/PULL`? — user36327, Oct 24 '19 at 05:54

score 0 · Answer 2 · answered Oct 24 '19 at 15:40

I see a couple of issues.

You are creating an array of 3 ints in size and then storing 9 ints into it int32_t info[3];
You are sending and receiving do not match regarding message parts.
- You are sending in this order (3 data's at end)
- byte_rows, byte_cols, byte_mat_type, byte_rows, byte_cols, byte_mat_type,byte_rows, byte_cols, byte_mat_type, data, data, data
- You are receiving in this order (data after each meta portion)
- byte_rows, byte_cols, byte_mat_type, data, byte_rows, byte_cols, byte_mat_type, data, byte_rows, byte_cols, byte_mat_type, data

How to communicate multiple images using ZeroMQ in C++?

2 Answers2

Solution ?