How to receive images from Raspberry Pi over ZeroMQ PUB/SUB in Python?

Question

I want to send images taken by Picamera on Raspberry Pi to my windows computer.

I wrote some code, as follows (they are simplified here), but it gets stuck in
frame = footage_socket.recv_string() in the client.py.

I don't get any error but it always gets stuck in the code like it freezes and can't go to the next line. The server.py works fine and prints 'test' continuously. If you look at the jpg_as_text, you can see encoded texts.

server.py :

import picamera
import socket
import threading
import zmq
import cv2
import base64
from picamera.array import PiRGBArray

if __name__ == "__main__":

    addr = 'ip_address'

    camera = picamera.PiCamera()                 # Camera initialization
    camera.resolution = (640, 480)
    camera.framerate = 7
    rawCapture = PiRGBArray(camera, size=(640, 480))

    # FPV initialization
    context = zmq.Context()
    footage_socket = context.socket(zmq.PUB)
    footage_socket.connect('tcp://%s:5555'%addr)
    print(addr)

    font = cv2.FONT_HERSHEY_SIMPLEX
    for frame in camera.capture_continuous( rawCapture,
                                            format         = "bgr",
                                            use_video_port = True ):
        image = frame.array
        print('test')
        image = cv2.resize(image, (640, 480))    # resize the frame
        encoded, buffer = cv2.imencode('.jpg', image)
        jpg_as_text = base64.b64encode(buffer)
        footage_socket.send(jpg_as_text)
        rawCapture.truncate(0)

client.py :

from socket import *
import sys
import time
import threading as thread
import tkinter as tk
import math
import os
import cv2
import zmq
import base64
import numpy as np


if __name__ == "__main__":

    context = zmq.Context()
    footage_socket = context.socket(zmq.SUB)
    footage_socket.bind('tcp://*:5555')
    footage_socket.setsockopt_string(zmq.SUBSCRIBE, np.unicode(''))

    font = cv2.FONT_HERSHEY_SIMPLEX

    while 100:
        try:
            frame = footage_socket.recv_string() # This line of code is the problem.

            print('next successfuly connected')
            img = base64.b64decode(frame)
            npimg = np.frombuffer(img, dtype=np.uint8)
            source = cv2.imdecode(npimg, 1)
            cv2.imshow("Stream", source)
            cv2.waitKey(1)

        except KeyboardInterrupt:
            break
        except:
            pass
    

Answer 1

Q : How to receive images from Raspberry Pi over ZeroMQ PUB/SUB in Python?

OBSERVATION :

There is no bug.

You shall use other data-acquisition strategy + setup some self-defensive parameters.

.recv_string()-method is called in a blocking-mode ( it does and will, even forever, block the code-execution, until anything plausible meets the rules to become deliverable

Using zmq.NOBLOCK flag permits you to avoid such blocking-mode + using a .poll()-method can help you design private event-driven loops' logic, that call for .recv( zmq.NOBLOCK ) just in cases, there indeed is something ready to get delivered.

SUB-side will receive nothing, unless properly subscribed to receive something, the default state -like with newspapers- is to receive nothing, unless explicitly subscribed to. The safest mode to subscribe to any content, as per the API documented strategy, is to subscribe to a zero-length string, using a .setsockopt( zmq.SUBSCRIBE, "" )-method to do so.

Last, but not least, if willing to do RPi-Win streaming, there might be a vise strategy, as it is in common of no value to enqueue/publish/transport/receive/dequeue any but the very latest frame, for which .setsockopt( zmq.CONFLATE, 1 ) is ready.

You may need more tweaking of resources, be it for boosting the .Context( nIOthreads )-instance performance, reserved Queue-depth, L3-stack parameters and many further possible enhancements.

Always do set .setsockopt( zmq.LINGER, 0 ) for you never know which versions will connect and what defaults might take place, here, with a chance to let your crashed instances of sockets hang forever (most often until the O/S reboot), which seems a bit wild, unhandled risk-factor for any production-grade software, doesn't it?

---

SOLUTION TIPS :

• Avoid a risk of having missed the unicode-conventions, which are different not matching one another between the Linux-side originator and the Windows O/S-side.

_{+
Since unicode objects have a wide range of representations, they are not stored as the bytes according to their encoding, but rather in a format called UCS (an older fixed-width Unicode format). On some platforms (OS X, Windows), the storage is UCS-2, which is 2 bytes per character. On most ix systems, it is UCS-4, or 4 bytes per character. The contents of the buffer of a unicode object are not encoding dependent (always UCS-2 or UCS-4), but they are platform dependent.
...
+
The efficiency problem here comes from the fact that simple ascii strings are 4x as big in memory as they need to be (on most Linux, 2x on other platforms). Also, to translate to/from C code that works with char, you always have to copy data and encode/decode the bytes. This really is horribly inefficient from a memory standpoint. Essentially, Where memory efficiency matters to you, you should never ever use strings; use bytes. The problem is that users will almost always use str, and in 2.x they are efficient, but in 3.x they are not. We want to make sure that we don’t help the user make this mistake, so we ensure that zmq methods don’t try to hide what strings really are.}

• Read more about latency-avoidance in ZeroMQ, if trying to stream the constant and a priori known imagery ( 640 x 480 x <colordepth> ) - conversions are expensive, turning a small-scale, low-res, low FPS RGB / IR picture into JPEG-file format just for transmission is meaningless if local LAN or a dedicated WLAN segment is used between the RPi and Win-device. Latency-motivated design may test and perhaps avoid any kind of compressing the data by using cPickle.dumps() or dill.dumps() but rather send data as compact as possible in binary-block-BLOB, most often enough to use aNumpyObject.data utility to send straight from <read-write buffer for 0x7fa3cbe3f8a0, size 307200, offset 0 at 0x7f632bb2cc30> or doing some binary mangling using struct.pack()/.unpack()-methods, if in a need to go beyond the numpy available .data-access trick. All given the .setsockopt( zmq.CONFLATE, 1 ) was activated on both sides, for avoiding any excessive depths of buffering live-streaming data.

• For both performance & latency reasons, you may avoid the PUB/SUB pair of Archetypes, as the ZeroMQ API v3.+ has moved the workload of the TOPIC-filtering onto the PUB-side, which is your weaker node ( while RPi has several cores and you may boost the .Context( nIOthreads )-instance on steroids, to have more power for I/O, yet the RPi has a fraction of GHz, compared to your Windows-side localhost, and robotic-tight control-loops may have already eaten up most of that for control ). Using PUSH/PULL would fit in quite the same way, for a 1-to-1 topology, plus having less processing and E2E-latency overheads due to handling avoided on the RPi side.

For .poll()-based, differently prioritised event-handlers, and a few remarks about seminal work of Mrs. Margaret HAMILTON and her MIT team, may like to read this & this.

Answer 2

You need to bind the server port not the client port.

server.py:

Change

footage_socket.connect('tcp://%s:5555'%addr) To footage_socket.bind('tcp://*:5555')

client.py:

Change footage_socket.bind('tcp://*:5555') To footage_socket.connect('tcp://%s:5555'%addr)