17

Good day,

I've searched for this but haven't come up with any responses. I wish to send a multi dimensional numpy array over a socket. Hence, I decided to convert it to a string:

However, it destroys the representation of the array:

>>> import numpy as np
>>> x = np.array([[0, 1], [2, 3]])
>>> xstring = x.tostring()
>>> print xstring

>>> print x
[[0 1]
 [2 3]]
>>> print xstring

>>> nparr = np.fromstring(xstring, dtype=np.uint8)
>>> print nparr
[0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0]

Is there anyway I can get the conversion to string to somehow, save the dimension of it?

Saullo G. P. Castro
  • 56,802
  • 26
  • 179
  • 234
ovfstack
  • 241
  • 1
  • 3
  • 9
  • 1
    You're almost doing it right - first - your `dtype` is not `np.uint8` - this loads the buffer as bytes (and it is serialized from `int64`). You need to use the correct `dtype` in `fromstring`, and then `.reshape()` the result to the original shape - in your case `(2, 2)`. If you're sending it e.g. over a socket, the shape needs to be provided separately. – Tomasz Gandor Apr 12 '18 at 12:25

4 Answers4

11

Try this example:- 

import socket
import numpy as np
from cStringIO import StringIO

class numpysocket():
    def __init__(self):
        pass

    @staticmethod
    def startServer():
        port=7555
        server_socket=socket.socket() 
        server_socket.bind(('',port))
        server_socket.listen(1)
        print 'waiting for a connection...'
        client_connection,client_address=server_socket.accept()
        print 'connected to ',client_address[0]
        ultimate_buffer=''
        while True:
            receiving_buffer = client_connection.recv(1024)
            if not receiving_buffer: break
            ultimate_buffer+= receiving_buffer
            print '-',
        final_image=np.load(StringIO(ultimate_buffer))['frame']
        client_connection.close()
        server_socket.close()
        print '\nframe received'
        return final_image

    @staticmethod
    def startClient(server_address,image):
        if not isinstance(image,np.ndarray):
            print 'not a valid numpy image'
            return
        client_socket=socket.socket()
        port=7555
        try:
            client_socket.connect((server_address, port))
            print 'Connected to %s on port %s' % (server_address, port)
        except socket.error,e:
            print 'Connection to %s on port %s failed: %s' % (server_address, port, e)
            return
        f = StringIO()
        np.savez_compressed(f,frame=image)
        f.seek(0)
        out = f.read()
        client_socket.sendall(out)
        client_socket.shutdown(1)
        client_socket.close()
        print 'image sent'
        pass

In this model client sends multidimensional ndarray to server. There are two functions startServer() and startClient(). startServer takes no arguments but startClient needs server address as well as the ndarray as arguments. First start Server and then start client. Server starts reading from buffer only after receiving the shutdown message from client.

Rashid Mv
  • 396
  • 3
  • 8
4

Indeed, .tostring only returns the raw data. This means you additionally need to send the shape and dtype of the array if these aren't known on the other side.

Maybe it's easier to serialize the array using Pickle:

import numpy as np
from cPickle import dumps, loads

x = np.array([[1, 2],[3, 4]], np.uint8)
print loads(dumps(x))
# [[1 2]
#  [3 4]]

Though for very small arrays the size overhead could be significant:

print len(x.tostring()), len(dumps(x))
# 4 171

For more on using Pickle, see here.

  • Pickle was extremely slow, and made my 2d array massive. It also occasionally crashed my program with some keyerror exceptions – ovfstack Oct 17 '14 at 03:32
  • @ovfstack: You're right. On Python 3 the overhead is negligible for large arrays and it runs only twice as slow as `.tostring()`. For Python 2 you should see a great improvement if you specify `protocol=2`, though it still seems to be 4 times slower than `.tostring()` in this case. –  Oct 17 '14 at 06:23
  • I just used the imencode function that opencv provides. It was also able to "compress" my array and imdecode was able to recover it with decent accuracy – ovfstack Oct 19 '14 at 07:04
0

This is a slightly improvised answer for ajsp answer using XML-RPC.

On the server-side when you convert the data, convert the numpy data to a string using the '.tostring()' method. This encodes the numpy ndarray as bytes string. On the client-side when you receive the data decode it using '.fromstring()' method. I wrote two simple functions for this. Hope this is helpful.

  1. ndarray2str -- Converts numpy ndarray to bytes string.
  2. str2ndarray -- Converts binary str back to numpy ndarray.
    def ndarray2str(a):
        # Convert the numpy array to string 
        a = a.tostring()

        return a

On the receiver side, the data is received as a 'xmlrpc.client.Binary' object. You need to access the data using '.data'.

    def str2ndarray(a):
        # Specify your data type, mine is numpy float64 type, so I am specifying it as np.float64
        a = np.fromstring(a.data, dtype=np.float64)
        a = np.reshape(a, new_shape)

        return a

Note: Only problem with this approach is that XML-RPC is very slow while sending large numpy arrays. It took me around 4 secs to send and receive a (10, 500, 500, 3) size numpy array for me.

I am using python 3.7.4.

Sudheer Raja
  • 151
  • 1
  • 8
0

I recently wanted to do a similar thing and tried to do everything on my own which takes up a lot of time and efforts. Turns out, a library already exists in python that does everything for you (maintains shapes and abstracts the tedious socket code)!

Check out - https://pypi.org/project/mlsocket/

Srikrishna Veturi
  • 1