1

I've read " Is shared readonly data copied to different processes for Python multiprocessing? " but the array described there is global. Is it possible to do the same with local arrays ?

Community
  • 1
  • 1
LBarret
  • 1,113
  • 10
  • 23
  • if you can use Cython, you can share the array between different threads only working with the pointers, [as shown here...](http://stackoverflow.com/a/20520295/832621) – Saullo G. P. Castro Jun 15 '14 at 11:04

1 Answers1

-1

I don't think so - but you can save stuff to a module variable. If you do this before the fork (and you are not on windows) it should work fine.

Eg

import mymodule

def somefunc(parameter):
    # do something with mymodule.var

# load/process local data
# save to module variable
mymodule.var = var
# now fork
p = multiprocessing.Pool(8)
p.map(somefunc, list_of_params)

If you use ipython you need to put somefunc in a module too (pickling functions in main doesn't seem to work).

robince
  • 10,826
  • 3
  • 35
  • 48
  • Doesn't work. Copy-on-write is invalidated by most reads due to the reference counter being incremented. – Will Jul 22 '13 at 09:45
  • But for large variables updating the reference counter will copy just the page containing the reference counter - not the whole (untouched) data. I work usually with large numpy arrays for which the above works well. – robince Jul 22 '13 at 20:47