0

I would like to access an existing NumPy array from a subprocess using the multiprocessing module without copying the array to a shared memory object. Apparently, multiprocessing.Array creates such a shared memory array, but I can't seem to be able to point the array to an existing numpy.ndarray object. This is critical, because the existing array can be quite large (up to a couple of GB), so I definitely need to avoid any copy operations.

Here's what I've tried so far:

import multiprocessing as mp
import numpy as np


def f(x, idx):
    """Dummy function to manipulate an array."""
    x[idx] = 999


a = np.array([1.2, 15.8, 10.3, 7.4, -44.9])
b = mp.Array("d", a)  # apparently this creates a copy of a in b
print("Original array:".rjust(28, " "), a)

f(a, 0)
print("Change a[0] in main process:".rjust(28, " "), a)

p = mp.Pool(1)

p.apply_async(f, args=(b, 4))
print("Change b[4] in subprocess:".rjust(28, " "), np.frombuffer(b.get_obj()))

Ideally, I'd like a and b refer to the same underlying numbers, but apparently this is not working. Interestingly, b is also not changed by p.apply_async(f, args=(b, 4)) - this is probably not related to the original question, but I'd still like to understand why.

cbrnr
  • 1,564
  • 1
  • 14
  • 28
  • Let me know if this thing worked :D – NoorJafri May 08 '18 at 14:07
  • I don't think this is a duplicate, since in this question `unshared_arr` and `arr` do not refer to the same address in memory, which is exactly what I'd like to do. – cbrnr May 08 '18 at 14:34
  • From the docs it looks like `mp.Array` creates a `ctypes` array in shared memory, and initializes it from values in `a`. It isn't storing or sharing a pointer to the `a` object. So `b` is not just a copy, it's not a numpy array; it's a different kind of data structure. Note in the docs it can intialized from a `range`, a `list` and `Array.array`. – hpaulj May 08 '18 at 16:34
  • Something went wrong with marking this as a duplicate - the wrong question is linked, could someone please correctly link to https://stackoverflow.com/questions/42037545/create-shared-memory-around-existing-array-python? – cbrnr May 09 '18 at 10:05

0 Answers0