3

The multiprocessing docs say it is better to inherit (from the ancestor process) than to pickle/unpickle a shared resource (and hence to avoid using pipes/queues).

However, it also says it is better to explicitly pass resources, rather than to share as global resources.

So, how do you pass a shared resource, e.g. to a map function?

This example shows that a shared global works (but is not explicit),

import multiprocessing.sharedctypes
import numpy as np

def task(i):
    global x
    x[i] = i

sharedresource = multiprocessing.sharedctypes.RawArray('b', 10)
x = np.frombuffer(sharedresource, dtype=np.uint8)

with multiprocessing.Pool(4) as pool:
    pool.map(task, range(10)) # implicitly modify x

print(x) # [0 1 2 3 4 5 6 7 8 9]

but explicit passing does not work (presumably pickling in some queue has broken the shared connection):

def task2(i, array):
    array[i] = 7

with multiprocessing.Pool(4) as pool:
    pool.starmap(task2, ((i,x) for i in range(10))) # explicit fail

print(x) # still [0 1 2 3 4 5 6 7 8 9]
benjimin
  • 4,043
  • 29
  • 48
  • https://stackoverflow.com/a/1721911/ suggests passing the resource as an `initargs` argument to `Pool` and writing an `initialize` function to assign it to a new global under each process. That might be more portable, but isn't terribly clean looking. – benjimin Mar 01 '18 at 05:54

0 Answers0