How to send custom Python objects through sockets without having a reference to object's class in other side of socket?

Question

I am trying to send a custom Python object from a client to a server, but the additional challenge of not having the reference to the object's class in the server. If you have the class declaration in the server script, then it reconstructs the Python object correctly. Otherwise, it just fails to reconstruct.

I have tried using pickle and jsonpickle to send the Python object (you can select both options through the OPTION variable). Both methods do not work after commenting out the class declaration.

Server

from socket import socket, gethostbyname, AF_INET, SOCK_DGRAM
import pickle

import jsonpickle

OPTION = 'pickle'
PORT_NUMBER = 5000
SIZE = 4000

# Required (but I don't want this!)
# class A:
#     ...

s = socket(AF_INET, SOCK_DGRAM)
s.bind((gethostbyname('0.0.0.0'), PORT_NUMBER))

(data,addr) = s.recvfrom(SIZE)
print(data)

if OPTION == 'pickle':
    output = pickle.loads(data)
    print(output)

elif OPTION == 'jsonpickle':
    output = jsonpickle.decode(data)
    print(output)

Client

import sys
from socket import socket, AF_INET, SOCK_DGRAM
import pickle

import jsonpickle

OPTION = 'pickle'
SERVER_IP   = '127.0.0.1'
PORT_NUMBER = 5000

class A:
    ...

a = A()

s = socket(AF_INET, SOCK_DGRAM)
s.connect((SERVER_IP, PORT_NUMBER))

if OPTION == 'pickle':
    s.send(pickle.dumps(a))
elif OPTION == 'jsonpickle':
    s.send(jsonpickle.encode(a).encode())

Outputs

These are the outputs when class A is in the server:

$ python server.py # jsonpickle
b'{"py/object": "__main__.A"}'
<__main__.A object at 0x7f4557a675e0>
$ python server.py # pickle
b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94.'
<__main__.A object at 0x7fcab9fc9bb0>

These are the outputs when class A is not part of the server:

$ python server.py # pickle
b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94.'
Traceback (most recent call last):
  File "/home/eduardo/Downloads/SO_Question/server.py", line 21, in <module>
    output = pickle.loads(data)
AttributeError: Can't get attribute 'A' on <module '__main__' from '/home/eduardo/Downloads/SO_Questi
on/server.py'>
$ python server.py # jsonpickle
b'{"py/object": "__main__.A"}'
{'py/object': '__main__.A'}

I looked into other approaches, like Pyro, but I am not too sure if these will work for my use case. Maybe ther are other tools that could help here. This server-client example is just a toy problem to a larger project I am working on: ChimeraPy. The goal would be for a server to distribute custom and unique code to clients. By using clients' computational resources to execute the custom code, using proxies, like Pyro, are not ideal as they don't distribute the computational load the same way as a cluster does.

Any help would be greatly appreciated. Thanks!

Tried:

Looked at the following SO questions: How to send classes trhough sockets in python, How to send an object over network, How to send classes through sockets in python?, and how to send objects through python?.
Tested sending a class with jsonpickle but it didn't help reconstruct
Looked deeper into jsonpickle documentation but couldn't identify anything to circumvent the issue

Hoping:

To find a solution for sending custom Python objects without reference in the other side of the socket

"Looked at multiple SO forums" which questions have you already looked at? — Michael M., Oct 26 '22 at 00:57
[1](https://stackoverflow.com/questions/64573569/how-to-send-classes-through-sockets-in-python) [2](https://stackoverflow.com/questions/47211732/how-to-send-an-object-over-network) [3](https://stackoverflow.com/questions/64573569/how-to-send-classes-through-sockets-in-python) [4](https://stackoverflow.com/questions/27542447/how-to-send-objects-through-python), searched multiple wordings of the following: "custom class python object sending through sockets" — Eduardo Davalos, Oct 26 '22 at 01:00
What do you _intend_ to do with the class on the `server` side? Are you planning to use only just the attributes or are you planning to call methods that the client `class A` may have declared? (if it's the latter you will need to figure out how to send the code over to the server, and that's certainly not recommended in an untrusted environment - can potentially [follow this approach](https://stackoverflow.com/questions/2626636/pickling-a-class-definition), another [related thread](https://stackoverflow.com/questions/34261379/how-to-recover-a-pickled-class-and-its-instances)) — metatoaster, Oct 26 '22 at 01:03
I am planning on using methods as well. I am aware of the danger of pickled python objects and sending them via socket. Still, I aim to integrate the socket programming component within a framework with restrictions to avoid sending malicious code. Will take a look at the references, thanks! — Eduardo Davalos, Oct 26 '22 at 01:14

score 0 · Accepted Answer · answered Oct 26 '22 at 03:51

Using the recommend links 1 and 2 by @metatoaster, I was able to achieve the desired behavior:

Server

from socket import socket, gethostbyname, AF_INET, SOCK_DGRAM
import pickle
import dill

import jsonpickle

OPTION = 'dill'
PORT_NUMBER = 5000
SIZE = 4000

# Required (but I don't want this!)
# class A:
#     ...

s = socket(AF_INET, SOCK_DGRAM)
s.bind((gethostbyname('0.0.0.0'), PORT_NUMBER))

(data,addr) = s.recvfrom(SIZE)
print(data)

if OPTION == 'pickle':
    output = pickle.loads(data)
    print(output)

elif OPTION == 'jsonpickle':
    output = jsonpickle.decode(data)
    print(output)

elif OPTION == 'dill':
    output = dill.loads(data)
    print(output)
    print(output.multiple(3))

Client

import sys
from socket import socket, AF_INET, SOCK_DGRAM
import pickle

import jsonpickle
import dill

OPTION = 'dill'
SERVER_IP   = '127.0.0.1'
PORT_NUMBER = 5000

class A:
    def __init__(self):
        self.num = 2

    def multiple(self, n):
        return self.num * n

a = A()

s = socket(AF_INET, SOCK_DGRAM)
s.connect((SERVER_IP, PORT_NUMBER))

if OPTION == 'pickle':
    s.send(pickle.dumps(a))
elif OPTION == 'jsonpickle':
    s.send(jsonpickle.encode(a).encode())
elif OPTION == 'dill':
    s.send(dill.dumps(a))

After running server.py and client.py, I get the following output:

$ python server.py 
b'\x80\x04\x95\x17\x02\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x0c_create_type\x94\x93\x94(h\
x00\x8c\n_load_type\x94\x93\x94\x8c\x04type\x94\x85\x94R\x94\x8c\x01A\x94h\x04\x8c\x06object\x94\x85\
x94R\x94\x85\x94}\x94(\x8c\n__module__\x94\x8c\x08__main__\x94\x8c\x08__init__\x94h\x00\x8c\x10_creat
e_function\x94\x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94(K\x01K\x00K\x00K\x01K\x02KCC\nd\x01|\x00
_\x00d\x00S\x00\x94NK\x02\x86\x94\x8c\x03num\x94\x85\x94\x8c\x04self\x94\x85\x94\x8c-/home/eduardo/Do
wnloads/SO_Question/client.py\x94h\x10K\rC\x02\x00\x01\x94))t\x94R\x94c__builtin__\n__main__\nh\x10NN
t\x94R\x94}\x94}\x94(\x8c\x0f__annotations__\x94}\x94\x8c\x0c__qualname__\x94\x8c\nA.__init__\x94u\x8
6\x94b\x8c\x08multiple\x94h\x12(h\x14(K\x02K\x00K\x00K\x02K\x02KCC\n|\x00j\x00|\x01\x14\x00S\x00\x94N
\x85\x94h\x18h\x19\x8c\x01n\x94\x86\x94h\x1bh(K\x10C\x02\x00\x01\x94))t\x94R\x94c__builtin__\n__main_
_\nh(NNt\x94R\x94}\x94}\x94(h#}\x94h%\x8c\nA.multiple\x94u\x86\x94b\x8c\x07__doc__\x94N\x8c\r__slotna
mes__\x94]\x94ut\x94R\x94)\x81\x94}\x94h\x17K\x02sb.'
<__main__.A object at 0x7fe8ec0656d0>
6

Success! Thanks for the help!

How to send custom Python objects through sockets without having a reference to object's class in other side of socket?

Server

Client

Outputs

1 Answers1

Server

Client