Sending numpy arrays over socket between Python versions, ascii error

Question

I need to send numpy arrays between Python2 and Python3 programs which run simultaneously. After doing some research, I've decided to send pickles over a socket connection.

I'm able to establish a connection and send strings back and forth, but trying to send numpy arrays results in a ascii error. Here is what my client/server are doing:

### Imports on both client and server ###
import socket as sc
import numpy
try:
    import cPickle as pickle
except ImportError:
    import pickle
pickle.HIGHEST_PROTOCOL = 2

### Python2 client ###
data= numpy.ones((1, 60))
sock.connect(SERVER_ADDR)

try:
    serialized_data = pickle.dumps(data, protocol=2)
except pickle.PicklingError as e:
    raise e

try:
    sock.send(serialized_data )
except sc.error as e:
    logging.error('Connection ended')

### Python3 server ###
"""
Given a client socket, and until the client has closed the session:
- receive input from client
- unpickle and print it
"""

while client != '':
    try:
        data = client.recv(4096)
    except sc.error as e:
        if e.errno != errno.ECONNRESET:
            print('Connection ended')
    else:
        try:
            unserialized_input = pickle.loads(input) ##### ERROR HERE #####
        except pickle.UnpicklingError as e:
            raise e
        else:
            print(unserialized_input)

### Error on server ###
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 6: ordinal not in range(128)

I trying searching for related questions, but couldn't find anything relevant except the solution suggested here, which doesn't work for me since I need to use Python2. I'm not sure how to continue. Any ideas?

Have you looked at https://docs.python.org/3/library/pickle.html#pickle.loads ? — pypypy, Aug 29 '17 at 15:46
TCP is a streaming protocol. How do you know you've received a full pickled message? You have to implement a protocol to ensure you've received the complete data. — Mark Tolonen, Aug 29 '17 at 18:06

Mark Tolonen · Accepted Answer · 2017-08-29T19:04:30.643

I reproduced your error running the server on Python 3 and client on Python 2. The following code is an example that can run the server and client on any combination of Pythons. pickle.loads on Python 3 defaults to decoding strings with ascii. To leave them as the original bytes sent, add encoding='bytes' as a parameter.

My protocol is "read everything from the client until the client closes the connection, then deserialize it". If you want to keep the connection open and send more than one serialized blob, you'll have to define a protocol that indicates the length of the serialized blob.

client:

import socket
import numpy

try:
    import cPickle as pickle
except ImportError:
    import pickle

sock = socket.socket()
data= numpy.ones((1, 60))
sock.connect(('localhost',8000))
serialized_data = pickle.dumps(data, protocol=2)
sock.sendall(serialized_data)
sock.close()
sock = socket.socket()
data= numpy.zeros((1, 60))
sock.connect(('localhost',8000))
serialized_data = pickle.dumps(data, protocol=2)
sock.sendall(serialized_data)
sock.close()

server:

from __future__ import print_function
import sys
import socket
import numpy

try:
    import cPickle as pickle
except ImportError:
    import pickle

s = socket.socket()
s.bind((b'',8000))
s.listen(1)
while True:
    c,a = s.accept()
    data = b''
    while True:
        block = c.recv(4096)
        if not block: break
        data += block
    c.close()
    if sys.version_info.major < 3:
        unserialized_input = pickle.loads(data)
    else:
        unserialized_input = pickle.loads(data,encoding='bytes')
    print(unserialized_input)

Output:

[[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
   1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
   1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.
   1.  1.  1.  1.  1.  1.]]
[[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
   0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
   0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
   0.  0.  0.  0.  0.  0.]]

Sending numpy arrays over socket between Python versions, ascii error

1 Answers1