Python multiprocessing object reference

Question

I don't get python multiprocessing module. I start a process with the function and pass and object as its parameter. As I understand it, it should be the exact copy of that object. But if I try to print the address of that object it is the same in every process. How it is possible? Shouldn't address be different in every process? And if it is the same object in every process how come that changes to it are not global but local for every process?

I have my object defined like this:

class MyClass():

    my_field = None

    def __init__():
        self.my_field = 0

And function which runs in separate process

def function_to_run_in_process(some_object):
    print some_object

The result for multiple processes is something like:

<__main__.MyClass instance at 0x7f276419e5f0>
<__main__.MyClass instance at 0x7f276419e5f0>
<__main__.MyClass instance at 0x7f276419e5f0>

and so on.

If I tried to change some field of the object inside the process like this:

def function_to_run_in_process(some_object, process_lock):
   process_lock.acquire()
   some_object.some_field = some_object.some_field + 1
   process_lock.acquire()
   print some_object, 'with some_field = ', some_object.some_field, 'at address: ', hex(id(some_object.some_field))

I get the result similar to this:

<__main__.MyClass instance at 0x7f276419e5f0>  with some_field = 1  at address  0xc5c428>
<__main__.MyClass instance at 0x7f276419e5f0>  with some_field = 1  at address  0xc5c428>
<__main__.MyClass instance at 0x7f276419e5f0>  with some_field = 1  at address  0xc5c428>

So, if the passed object is just a copy, why there are same addresses not only for the object, but even for its field? And if they are the same why the change to the field is not visible ?

Why would you acquire a lock if the object is just a copy? There would be no risk for simultaneous access then. — Jacques de Hooge, Jul 09 '15 at 09:07

MattH · Accepted Answer · 2015-07-09T09:21:39.600

How it is possible?

Each process has it's own Virtual Address Space

Shouldn't address be different in every process?

No. The child processes are inheriting the VAS of their parent. See clone and fork.

And if it is the same object in every process how come that changes to it are not global but local for every process?

The process memory pages in the VAS will be set to COPY ON WRITE. As long as the page isn't changed, they'll all point to the same physical memory, however if any changes are made, the VAS will for that page will be mapped to a different location.

Here's an article on process memory management: http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory/

If you want to share state between processes, you can share memory or pass messages. The multiprocessing module can create python objects in shared memory pages, this is described here

However, it is best avoided, especially if all of the above was news to you.

Python multiprocessing object reference

1 Answers1

Linked