How can I send PNG image over TCP sockets using ZeroMQ?

Question

I have a Keras model working on Python and I want to send frames to this model from Unity 3D camera. I can easily transport strings among them by using an external library. This external library sends a byte array to Python. So, I converted Unity's camera frames to a byte array. However, I do not know how to read the sent images (byte array) on Python.

In order to handle this problem, I have tried some Sender-Receiver codes using only Python. However, those did not work.

Here are the codes:

Sender :

import zmq
import base64


context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")

f = open("sample.png", 'rb')
bytes = bytearray(f.read())
strng = base64.b64encode(bytes)
socket.send(strng)
f.close()

Receiver :

import zmq
import base64


context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")

import base64
message = socket.recv()
f = open("sample.png", 'wb')
ba = bytearray(base64.b64decode(message))
f.write(ba)
f.close()

Is that a good way for sending frames from Unity to Python? Or are there other ways? How can I handle my problem? What is wrong with the codes above?

Your question mentions *"frames"*, *"byte arrays"* and *"strings"*, but your code seems to be reading/writing PNG images from/to disk. Could you be a bit more explicit about exactly what you start with at the sending end (maybe including the first few bytes of data) and exactly what you want to end up with at the receiving end. I am guessing neither is a disk file like your code. Thank you. — Mark Setchell, Jun 06 '20 at 16:06
In the end, I want to send frames from my game (Unity) to Python and analyze the frames using my Keras model. Then, I will send the results (commands) to Unity but this part is easier. — Terminou, Jun 06 '20 at 16:39
It's still not clear to me what Python data-structure your *"frame"* is. Is it a list? Or a string? Or a Numpy array? Please can you give a line or two of code that makes a red frame of say 128 pixels wide by 64 pixels tall? Or anything tangible. Thank you. — Mark Setchell, Jun 06 '20 at 16:45
"frame" will be a byte array since anything that can be converted to bytes can be transferred via ZeroMQ. Then in python, I need to decode the bytes array into an image. — Terminou, Jun 06 '20 at 17:56
Saying it's a byte array is like saying it's information, or it's some bytes. What are the bytes? What do they mean? Are the first 4 bytes `\x89PNG` meaning it's a PNG image? Or are they `\xff\xd8` meaning it's a JPEG? Or are they `\xff\x00\x00` meaning it's a red pixel? — Mark Setchell, Jun 06 '20 at 19:00
Why are you base64-encoding the image? You will just make it 30% bigger. — Mark Setchell, Jun 06 '20 at 19:02
As I said, I found the code on SO. I do not have any idea how to decode the byte array. However, here is how I produce byte arrays in Unity (C#): http://prntscr.com/sv18su — Terminou, Jun 06 '20 at 19:08

score 0 · Accepted Answer · edited Dec 08 '20 at 16:02

Q : "Is that a good way for sending frames from Unity to Python?"

I dare to say.

_{However, I do not know how to read the sent images (byte array) on Python.}

Why would ever one use a tool to "encode" something,
without having the matching tool to "decode"?

Q : "Or are there other ways?"

Oh sure there are.

Q : "How can I handle my problem?"

Best learn such tools, that you can safely use, both for the encode & the decode phases. Plenty of such available as Python standard modules - numpy with pil/pillow or cv for image conversions, dill or pickle for SER/DES or even a struct.

Having used these for ages with ZeroMQ, you may be sure to go in the right direction using safe tools.

Q : "What is wrong with the codes above?"

This will never fly.

The "Sender",
having been proposed as a REP-instance can never start with a call to the .send()-method.
Never.

The "Receiver",
having been proposed as a REQ-instance can never start with a call to the .recv()-method.
Never.

Sounds complicated? Yes, the choice of REQ/REP was not mine.

ZeroMQ REQ/REP Formal Scalable Communication Archetype is one of the most complicated ( operating an internal distributed Finite-State-Automaton (dFSA), actually 1:N of them, having not a risk, but a solid certainty of un-salvageably self-deadlocking any one of these N dFSA and the story ends... )

Do not get me wrong, I love ZeroMQ since ever and admire Pieter HINTJENS' & Martin SUSTRIK's immense amounts of invention & performance polishing they've breathed into the ZeroMQ's Zen of Zero, yet I try you to warn it takes some time to master the Zen start to end.

The link above has a full solution of using rather the PUB/SUB for image-sending plus some performance tweaking hints for further thinking.

How can I send PNG image over TCP sockets using ZeroMQ?

1 Answers1