multiprocessing Pool without return value?

Question

I am trying to use a multiprocessing Pool without a return value for parallel calculation. It could be faster if there's no need to return and retrieve values from the subprocess. Is there a way to do that?

Here is a simple example:

from multiprocessing import Pool

def fun(a):
    # do something.. 
    a["1"]=100

a={
   "1":12
   }
multi = [a] * 10
p = Pool(4)
p.map(fun, multi)
data = [a["1"] for a in multi]
print(data)
>>> [12, 12, 12, 12, 12, 12, 12, 12, 12, 12]

[fun(a) for a in multi]
data = [a["1"] for a in multi]
print(data)
>>> [100, 100, 100, 100, 100, 100, 100, 100, 100, 100]

Does anybody know why? And is there a solution for that?

The "return and retrieve values from subprocess" part is unlikely to be a bottleneck, that is a very fast operation — bphi, Dec 13 '17 at 15:49
For example, if no need to retrieve values, would it be more faster to use `p.map_async` instead of `p.map`? — Cheng, Dec 13 '17 at 15:54
See https://stackoverflow.com/questions/35908987/python-multiprocessing-map-vs-map-async — bphi, Dec 13 '17 at 15:57
[PyBay 2017 Keynote on Concurrency](https://www.youtube.com/watch?v=9zinZmE3Ogk) — wwii, Dec 13 '17 at 16:01
Threads can/do share state because they run in the same process. It takes extra *work* to share *state* between multiple processes. ... [Shared State in Python multiprocessing Processes](https://stackoverflow.com/q/30264699/2823755) .. there are others. — wwii, Dec 13 '17 at 17:05
Thanks for wwli. I think the answer below by mata make it clear. It can/do share state by using proxy objects. — Cheng, Dec 14 '17 at 05:36

mata · Accepted Answer · 2017-12-14T09:04:43.850

Your function fun

def fun(a):
    # do something.. 
    a["1"]=100

changes a mutable argument a. However when you call this using p.map(fun, multi) each item in the multi list is pickled, sent to a worker process and mutated there. This can't have any effect on the original items in the list in the calling process.

You can create data structures that can be shared between processes, so called proxy objects, using managers. You'd have to create 10 shared dictionaries. In your example you only have one dictionary, the list contains 10 references to it, data = [a["1"] for a in multi] will always only contain the same value because a is always the same object.

So this should work:

from multiprocessing import Pool, Manager
import random

def fun(a):
    # to show that the dictionaries are different
    a["1"] = random.random()

if __name__ == '__main__':
    m = Manager()
    p = Pool(4)
    multi = [m.dict() for _ in range(10)]
    p.map(fun, multi)
    data = [a["1"] for a in multi]
    print(data)

Note that multi = m.list([a] * 10) or similar would not work, because only list access is synchronized, not updates of the contained elements. But all of this creates additional IPC overhead and probably will be worse then just using the return value of the function if you can.

Thanks! It did work. I learned something new about proxy objects. It is very helpful and inspiring! — Cheng, Dec 14 '17 at 05:20
I had to say that calculation goes extremely slow compared with the regular dictionary list, when data becomes large for multi processes. — Cheng, Dec 14 '17 at 12:50
That's what I ment with _IPC overhead_, each access behind the scene needs to be synchronized, data sent to a different process and so on. If you have to exchange large amounts of data between your processes, then multiprocessing won't really help much. If your program can be changed to work with [shared memory](https://docs.python.org/3/library/multiprocessing.html#sharing-state-between-processes), that may help. — mata, Dec 14 '17 at 12:58

multiprocessing Pool without return value?

1 Answers1