How to share the memory with multiprocessing.Pool's workers, without using global variables?

Question

There are two functions:

def tibidam(..., foo_dyn, index):
    print("(" + str(index) + ") B:", foo_dyn)

    for i in range(...):
        for j ...
            if j not in foo_dyn:
                foo_dyn[ep] = j

    print("(" + str(index) + ") A:", foo_dyn)

def tata(..., foo):
    foo_dyn = Array('i', len(foo))
    
    foo_dyn = foo

    with Pool(processes=4) as pool:
        pool.starmap(tibidam, [(..., foo_dyn, i) 
            for i in range(4)])
    
    return foo

Output (formatted):

foo  : [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(0) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(1) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(2) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(3) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(0) A: [27, 1,  2,  3,  64, 5,  6,  7,  80, 9,  10, 11]
(2) A: [0,  1,  64, 3,  4,  5,  13, 7,  8,  9,  92, 11]
(3) A: [0,  1,  2,  31, 4,  5,  6,  73, 8,  9,  10, 18]
(1) A: [0,  18, 2,  3,  4,  27, 6,  7,  8,  99, 10, 11]
...

Expected output (formatted):

foo  : [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(0) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(1) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(2) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(3) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(0) A: [27, 1,  2,  3,  64, 5,  6,  7,  80, 9,  10, 11]
(2) A: [27, 1,  55, 3,  64, 5,  13, 7,  80, 9,  92, 11]
(3) A: [27, 1,  55, 31, 64, 5,  13, 73, 80, 9,  92, 18]
(1) A: [27, 87, 55, 31, 64, 88, 13, 73, 80, 99, 92, 18]
...

How I can change foo_dyn in all workers, whenever foo_dyn changes? It seems, that pool.starmap(...) creates a copy of the foo_dyn, for each process... No, I want pass foo_dyn to the pool only once. But, again, without using global variables, at all.

AFAIK, multiprocessing.Pool supports initializer and initargs arguments: I can write own initializer:

_init(foo):
    global foo_dyn

    foo_dyn = foo

, but it uses global variable foo_dyn (by the way, using _init function doesn't solve the problem). In passing, I saw a few questions, with almost the same trouble. However, all solutions were associated with using global variables.

Shared memory : multiprocessing module provides Array and Value objects to share data between processes. Array: a ctypes array allocated from shared memory. Value: a ctypes object allocated from shared memory. — Joe, Mar 15 '20 at 06:12
https://docs.python.org/3.8/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory — Joe, Mar 15 '20 at 06:13
https://docs.python.org/3.8/library/multiprocessing.html#sharing-state-between-processes — Joe, Mar 15 '20 at 06:14
Oh, @Joe, thanks you! :) I've seen that, but I'm more interested in a solution compatible with older Python versions. For example, it can be implemented with using multiprocessing.Process class. But, due to inefficiency, for my case, of this approach, I want use multiprocessing.Pool class (without using global variables). — Kassi, Mar 15 '20 at 11:36
`Value` and `Array` are available in 2.7 https://docs.python.org/2.7/library/multiprocessing.html#sharing-state-between-processes — Joe, Mar 15 '20 at 12:42
https://stackoverflow.com/questions/39322677/python-how-to-use-value-and-array-in-multiprocessing-pool — Joe, Mar 15 '20 at 12:43
https://stackoverflow.com/questions/1675766/combine-pool-map-with-shared-memory-array-in-python-multiprocessing — Joe, Mar 15 '20 at 12:45
@MaximTemny: Can’t you pass objects of the `multiprocessing` types as function arguments? — Davis Herring, Mar 17 '20 at 02:44
@DavisHerring, what do you mean? I pass `foo_dyn` (`multiprocessing.Array`) to `tibidam(...)`. — Kassi, Mar 17 '20 at 16:37
@MaximTemny: Sorry, I phrased that very poorly. I mean to say that it *should* work to pass such objects, and you should phrase your question as “Why isn’t this working?” with the usual complete reproducer. (For example, your code shows `foo_dyn` being reassigned to be just `foo`, which is surely wrong.) — Davis Herring, Mar 17 '20 at 21:15
@DavisHerring, oh, exactly... Didn't pay attention to it... But, differently, using `multiprocessing.Array`, I couldn't assign `foo` to `foo_dyn`, because it... Well, thank you. :> — Kassi, Mar 17 '20 at 21:25

score 1 · Accepted Answer · answered Oct 13 '20 at 01:04

I found the solution, without using the global variables:

from multiprocessing import Pool, Manager

def tibidam(..., foo_dyn, index):
    for i in range(...):
        for j ...
            if j not in foo_dyn:
                foo_dyn[ep] = j

def tata(..., foo):
    foo_dyn = Manager().list(foo)

    with Pool(processes=4) as pool:
        pool.starmap(tibidam, [(..., foo_dyn, i)
            for i in range(4)])

    return foo_dyn

Thank you all! :>

How to share the memory with multiprocessing.Pool's workers, without using global variables?

1 Answers1