Python multiprocessing: object passed by value?

Question

I have been trying the following:

from multiprocessing import Pool

def f(some_list):
    some_list.append(4)
    print 'Child process: new list = ' + str(some_list)
    return True

if __name__ == '__main__':

    my_list = [1, 2, 3]
    pool = Pool(processes=4)
    result = pool.apply_async(f, [my_list])
    result.get()

    print 'Parent process: new list = ' + str(my_list)

What I get is:

Child process: new list = [1, 2, 3, 4]
Parent process: new list = [1, 2, 3]

So, it means that the my_list was passed by value since it did not mutate. So, is the rule that it is really passed by value when passed to another process? Thanks.

score 9 · Accepted Answer · answered Jan 09 '15 at 20:22

As André Laszlo said, the multiprocessing library needs to pickle all objects passed to multiprocessing.Pool methods in order to pass them to worker processes. The pickling process results in a distinct object being created in the worker process, so that changes made to the object in the worker process have no effect on the object in the parent. On Linux, objects sometimes get passed to the child process via fork inheritence (e.g. multiprocessing.Process(target=func, args=(my_list,))), but in that case you end up with a copy-on-write version of the object in the child process, so you still end up with distinct copies when you try to modify it in either process.

If you do want to share an object between processes, you can use a multiprocessing.Manager for that:

from multiprocessing import Pool, Manager

def f(some_list):
    some_list.append(4)
    print 'Child process: new list = ' + str(some_list)
    return True

if __name__ == '__main__':

    my_list = [1, 2, 3]
    m = Manager()
    my_shared_list = m.list(my_list)
    pool = Pool(processes=4)
    result = pool.apply_async(f, [my_shared_list])
    result.get()

    print 'Parent process: new list = ' + str(my_shared_list)

Output:

Child process: new list = [1, 2, 3, 4]
Parent process: new list = [1, 2, 3, 4]

Thanks. What if I just make my f function return the modified object? Will I lose a lot in efficiency comparing to using Manager? — jazzblue, Jan 09 '15 at 20:31
@jazzblue If your use-case allows you to just modify the list in the child and return it to the parent, use that instead of the `Manager`. With the `Manager` object, every single time you access the shared list, you're actually making an IPC call to a `Manager` process. This is *much* slower than accessing the list natively. By returning the object to the parent, you do take the one-time IPC hit of passing the list object back between the processes, but unless you're dealing with a *huge* list, that's probably not going to outweigh the cost of all the IPC required with a shared list. — dano, Jan 09 '15 at 20:37

André Laszlo · Answer 2 · 2015-01-09T20:20:22.847

6

The multiprocessing library serializes objects using pickle to pass them between processes.

This ensures safe inter-process communication, and two processes can use the "same" objects without using shared memory.

edited Jan 09 '15 at 20:20

answered Jan 09 '15 at 20:14

André Laszlo

15,169
3
63
81

1

You should replace every instance of "thread" above with "process". – dano Jan 09 '15 at 20:16

Python multiprocessing: object passed by value?

2 Answers2

Linked