Sharing numpy arrays between multiple processes without inheritance

Question

I would like to share numpy arrays between multiple processes. There are working solutions here. However they all pass the arrays to the child process through inheritance, which does not work for me because I have to start a few worker processes beforehand and I don't know how many arrays I'm going to deal with later on. Is there any way to create such arrays after the process is started and pass these arrays to the processes via queues?

Btw for some reason I'm not able to use multiprocessing.Manager.

@Eric When you create a process, internally it does the `fork` system call and all the program states of the parent process are inherited by the child process, including the shared memory handles. An example of avoiding pass-by-inheritance is here: http://stackoverflow.com/questions/9908781/sharing-a-result-queue-among-several-processes — shaoyl85, Jan 16 '16 at 07:27
Using pyzmq to exchange the arrays is quit fast and flexible : https://github.com/zeromq/pyzmq/blob/master/examples/serialization/serialsocket.py — Niemerds, Jan 16 '16 at 09:35
@Niemerds Thank you! If I send arrays using multiprocessing.Queue, it calls pickle dumps and then send the dumped string via pipe. Does the use of pyzmq speed up any of these steps? — shaoyl85, Jan 16 '16 at 18:19
Yes definitly. The pickle function converts the array data to the binary format used by the (un-)pickle operation and vice-versa on the receiver side, while the pyzmq method sends/receives the numpy data unmodified using the buffer interface. Therefore unnecessary copies or conversions of the data on both sides are avoided. — Niemerds, Jan 16 '16 at 21:18
@skrrgwasme, I'm not sure, it was some kind of file system restriction. — shaoyl85, Apr 25 '16 at 01:41

score 3 · Answer 1 · answered Apr 06 '19 at 03:53

You should use shared memory, which exactly solve your use case. You keep memory read/write speed, and all processes can read and write in the array in shared memory without incurring any serialization or transport cost.

Below is the example from the official python doc:

>>> # In the first Python interactive shell
>>> import numpy as np
>>> a = np.array([1, 1, 2, 3, 5, 8])  # Start with an existing NumPy array
>>> from multiprocessing import shared_memory
>>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
>>> # Now create a NumPy array backed by shared memory
>>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
>>> b[:] = a[:]  # Copy the original data into shared memory
>>> b
array([1, 1, 2, 3, 5, 8])
>>> type(b)
<class 'numpy.ndarray'>
>>> type(a)
<class 'numpy.ndarray'>
>>> shm.name  # We did not specify a name so one was chosen for us
'psm_21467_46075'

>>> # In either the same shell or a new Python shell on the same machine
>>> import numpy as np
>>> from multiprocessing import shared_memory
>>> # Attach to the existing shared memory block
>>> existing_shm = shared_memory.SharedMemory(name='psm_21467_46075')
>>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example
>>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf)
>>> c
array([1, 1, 2, 3, 5, 8])
>>> c[-1] = 888
>>> c
array([  1,   1,   2,   3,   5, 888])

>>> # Back in the first Python interactive shell, b reflects this change
>>> b
array([  1,   1,   2,   3,   5, 888])

>>> # Clean up from within the second Python shell
>>> del c  # Unnecessary; merely emphasizing the array is no longer used
>>> existing_shm.close()

>>> # Clean up from within the first Python shell
>>> del b  # Unnecessary; merely emphasizing the array is no longer used
>>> shm.close()
>>> shm.unlink()  # Free and release the shared memory block at the very end

For a real use case as yours, you would need to pass the name shm.name using a Pipe or any other multi-processing communication mechanism. Note that only this tiny string will need to be exchanged between processes; the actual data stays in the shared memory space.

score 0 · Answer 2 · answered Jun 23 '18 at 23:38

Depending on your exact use case, using np.memmap for the arrays you want to transfer can be a good approch. The data will be on disk but it's used like a standard array, and only the "header" data is pickled in the Queue so it's very fast.

See https://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html

Sharing numpy arrays between multiple processes without inheritance

2 Answers2