Object Transmission in Python using Pickle

Question

I have the following class, a Point object

class Point:
    def __init__(self):
        pass

    def __init__(self, x, y):
        self.x = x
        self.y = y

And I have a server (Uses UDP)

# Server side
import socket
import pickle

host = "localhost"
port = 10000

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind((host, port))

data = s.recvfrom(1024)
print(data)

And my client side is:

import socket
import pickle
from Point import *

host = "localhost"
port = 10000
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

p = Point(10, 20)
a = pickle.dumps(p)

s.sendto(a, (host, port))

In the server side, whenever I get the p and print it, I get the following (b'\x80\x03cPoint\nPoint\nq\x00)\x81q\x01}q\x02(X\x01\x00\x00\x00xq\x03K\nX\x01\x00\x00\x00yq\x04K\x14ub.', ('127.0.0.1', 55511))

How can I get the object, instead of this?

...use `pickle.loads`? What does "it does not work" mean? Does it throw an error? Does it crash your PC? Does it return a Circle instead of a Point? — Aran-Fey, Feb 25 '18 at 13:41
@Aran-Fey I did so, but I still can't load create the object, I updated my question to show what I have tried — tony9099, Feb 25 '18 at 13:47
Why do you have **two** `__init__` methods? The first definition is a dud, replaced by the second. — Martijn Pieters, Feb 25 '18 at 13:50
I **strongly** want to caution against using Pickle in a client-server application, not unless you want to open yourself to having the process hijacked. Pickles allow for arbitrary code execution. — Martijn Pieters, Feb 25 '18 at 13:52
@MartijnPieters thanks ! I updated the question. Still, out of curiosity, want to know how to make this happen. — tony9099, Feb 25 '18 at 13:54
@tony9099 you still haven't explained what "but it does not work" means... do you get an exception for instance? — Jon Clements, Feb 25 '18 at 13:59
Now I am not clear what you are asking. You surely already read the documentation and used `pickle.loads()`. — Martijn Pieters, Feb 25 '18 at 14:05
Works now, thanks guys. I also posted the answer for future references. — tony9099, Feb 25 '18 at 14:05

Martijn Pieters · Answer 1 · 2018-02-25T14:56:12.280

1

First, a caveat. Pickles allow for arbitrary code execution. Do not use this to accept arbitrary connections, and preferably use cryptography to ensure you are only exchanging trusted data. Even then, consider using a safer exchange format.

Next, take into account that UDP packets are limited in size. You need to make sure your pickle data is small enough to fit in a UDP packet (the maximum payload size is 65507 bytes). At least when you receive the packet, you'll know you have all the data. Use 65535 as the buffer size to ensure large packets can be fully received.

On sending, make sure you don't cross the size limits:

MAX_UDP_SIZE = 65507  # https://en.wikipedia.org/wiki/User_Datagram_Protocol

a = pickle.dumps(p)
if len(a) > MAX_UDP_SIZE:
    raise ValueError('Message too large')
s.sendto(a, (host, port))

and on the other side, use pickle.loads() to turn the pickle data stream back into an object:

UDP_MAX = 2 ** 16 - 1

data, addr = s.recvfrom(UDP_MAX)
object = pickle.loads(data)

I urge you strongly to at the very least verify that addr is trusted, or you leave yourself open to executing arbitrary code. 65507 bytes is ample space to send a pickle that takes control of your process.

If you need to send more data, then you'll need to use TCP instead of UDP, because you'll have to send across data in a specific order, spread across multiple packets, and need for all packets to arrive on the receiving end; TCP provides that layer of reliability. At that point you'd have to prefix your pickle with a fixed number of bytes encoding the size of the pickle, so you can ensure you read that same amount of data again on the other side.

edited Feb 25 '18 at 14:56

answered Feb 25 '18 at 14:14

Martijn Pieters

1,048,767
296
4,058
3,343

The comments about framing are wrong. The OP uses UDP. The framing is the datagram. The correct amount of data to handle is already known. – Jean-Paul Calderone Feb 25 '18 at 14:32
@Jean-PaulCalderone: made it UDP specific now, thanks for pointing that out. – Martijn Pieters Feb 25 '18 at 14:56
Chances of getting 2 ** 16 byte UDP packets over the public internet is pretty small. You might pull it off over a LAN. The ease of forging a sender address on UDP is pretty great so don't rely on the reported sender address for access control. For crypto, it's very easy to get things wrong. You almost certainly need to use an existing, widely accepted solution. Consider DTLS with certificates or *perhaps* NaCL SecretBox. – Jean-Paul Calderone Feb 25 '18 at 16:01
(But really there are tons of alternatives to Pickle, just don't use it.) – Jean-Paul Calderone Feb 25 '18 at 16:02
Python has good [cryptography support](https://cryptography.io/en/latest/) but you do need to know what you are doing, yes. – Martijn Pieters Feb 25 '18 at 16:30
I'd love to hear what is not helpful or wrong about my answer, to deserve a downvote. That way I can improve my answer! – Martijn Pieters Feb 26 '18 at 18:48

score -2 · Answer 2 · answered Feb 25 '18 at 13:50

-2

Pickle is not suitable for this purpose. Consider this warning:

Warning The pickle module is not secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

Your protocol lacks any kind of access control so your processes are vulnerable to arbitrary remote code execution attacks.

Consider a real network protocol instead. There are many to choose from. For example, Cap' Proto.

answered Feb 25 '18 at 13:50

Jean-Paul Calderone

47,755
6
94
122

2

While true, this is a separate issue and not an answer to the question. – Martijn Pieters Feb 25 '18 at 13:51
Ignoring the security consequences here is irresponsible. – Jean-Paul Calderone Feb 25 '18 at 14:30

score -2 · Answer 3 · answered Feb 25 '18 at 14:05

-2

Solution is to used pickle.loads as below on the server side

d = s.recvfrom(1024)
p1 = Point(1,1)
p1 = pickle.loads(d[0])
print(str(p1.x))

However, as suggested by many, pickle might not be the best solution for such scenarios.

answered Feb 25 '18 at 14:05

tony9099

4,567
9
44
73

2

Reading a fixed 1024 bytes will cause issues if your data sent is larger than that size. UDP has size limits you need to account for. – Martijn Pieters Feb 25 '18 at 14:26

Object Transmission in Python using Pickle

3 Answers3

Linked