How to IPC between node and python using posix message queue?

Question

I'm trying to write a "hello world" of IPC between a node js and a python3 application. At the moment I'm having a parse error when a new message arrives the Python app. The codes are:

Python:

from ipcqueue import posixmq
from time import sleep

q1 = posixmq.Queue('/fila1')

while True:

    while q1.qsize() > 0:
        print('p1: Recebi na minha fila: ' + str(q1.get()))

    sleep(0.5)

Node:

const PosixMQ = require('posix-mq')

var mq = new PosixMQ();
mq.open({
    name: '/fila1',
    create: true,
    mode: '0777',
    maxmsgs: 10,
    msgsize: 11
});
mq.push("hello world")
mq.close();

When the second app sends the message, the python app fails with:

File "../test-fila1.py", line 9, in
print('p1: Recebi na minha fila: ' + str(q1.get()))   File "/usr/lib/python3.7/site-packages/ipcqueue/posixmq.py", line 174, in
get
return self._serializer.loads(data)   File "/usr/lib/python3.7/site-packages/ipcqueue/serializers.py", line 14,
in loads
return pickle.loads(data) KeyError: 101
[2]- Exit 1 python3 ../test-fila1.py [3]+ Done node index.js

EDIT
So, I decided to change the question from "KeyError: 101 when loading ipc message between node and python apps" to "How to IPC between node and python using posix message queue?" because I just noticed that the sample code I posted generates different errors everytime I run them. Now the error that is happening is (however, no code was changed):

Traceback (most recent call last): File "snippets/test-fila1.py",
line 9, in
print('p1: Recebi na minha fila: ' + str(q1.get())) File "/usr/lib/python3.7/site-packages/ipcqueue/posixmq.py", line 174, in
get
return self._serializer.loads(data) File "/usr/lib/python3.7/site-packages/ipcqueue/serializers.py", line 14,
in loads
return pickle.loads(data)
_pickle.UnpicklingError: unpickling stack underflow

try sending an object from node.js rather than raw string e.g. mq.push({'data': 'hello world'}); — Loveen Dyall, Jul 21 '19 at 19:34
the function doesn't accept it: `TypeError: First argument must be a node::Buffer or v8::String` — Fnr, Jul 23 '19 at 14:25
try stringifying a JSON object message. e.g. message = {'data': 'hello world'} mq.push(JSON.stringify(message)) — Loveen Dyall, Jul 23 '19 at 16:09
yup, it also results in the same error that was happening: `_pickle.UnpicklingError: invalid load key, '{'` — Fnr, Jul 23 '19 at 17:00

score 8 · Accepted Answer · answered Aug 04 '19 at 14:59

If you look the __init__ of the Queue

class Queue(object):
    """
    POSIX message queue.
    """

    def __init__(self, name, maxsize=10, maxmsgsize=1024, serializer=PickleSerializer):

As you can see the default serializer is PickleSerializer, which means it assumes the data coming on the queue is pickled, while you are sending raw data from nodejs. The fix is simple, use the RawSerializer

from ipcqueue import posixmq
from time import sleep
from ipcqueue.serializers import RawSerializer
q1 = posixmq.Queue('/fila1', serializer=RawSerializer)

while True:

    while q1.qsize() > 0:
        print('p1: Recebi na minha fila: ' + str(q1.get()))

    sleep(0.5)

How to IPC between node and python using posix message queue?

1 Answers1