Python multithreading doesn't use common memory when the parent object is created inside a process

Question

I'm trying to implement a thread which will run and do a task in the background. Here's my code which gives expected output.

Code 1:

from time import sleep
from threading import Thread

class myTest( ):
    def myThread( self ):
        while True:
            sleep( 1 )
            print self.myList.keys( )
            if 'exit' in self.myList.keys( ):
                break
        return

    def __init__( self ):
        self.myList = { } 
        self.thread = Thread( target = self.myThread, args = (  ) )
        self.thread.start( )
        return

    def myFun( self ):
        i = 0
        while True:
            sleep( 0.5 )
            self.myList[ i ] = i
            i += 1
            if i > 5 :
                self.myList[ 'exit' ] = ''
                break
        return

x = myTest( )
x.myFun( )

Output 1:

[0]
[0, 1, 2]
[0, 1, 2, 3, 4]
[0, 1, 2, 3, 4, 5, 'exit']

The moment I create a multi-process environment and create this object in the new sub-process, the thread is unable to access the common memory and the dictionary myList remains blank in the thread. Here is the code.

Code 2:

from time import sleep
from threading import Thread
from multiprocessing import Process

class myTest( ):
    def myThread( self ):
        while True:
            sleep( 1 )
            print "myThread", self.myList.keys( )
            if 'exit' in self.myList.keys( ):
                break
        return

    def __init__( self ):
        self.myList = { } 
        self.thread = Thread( target = self.myThread, args = (  ) )
        self.thread.start( )
        self.i = 0
        return

    def myFun( self ):
        self.myList[ self.i ] = self.i
        self.i += 1
        if self.i > 5 :
            self.myList[ 'exit' ] = ''
            return 'exit'
        print 'myFun', self.myList.keys( )
        return

class myProcess( Process ):
    def __init__( self ):
        super( myProcess, self ).__init__( )
        self.myObject = myTest( )
        return

    def run(self):
        while True:
            sleep( 0.5 )
            x = self.myObject.myFun( )
            if x == 'exit':
                break
        return

x = myProcess( )
x.start( )

Output 2:

myFun [0]
myThread []
myFun [0, 1]
myFun [0, 1, 2]
myThread []
myFun [0, 1, 2, 3]
myFun [0, 1, 2, 3, 4]
myThread []
myThread []
myThread []
... ... ...
... ... ...

In the 1st code, the object is created inside a Python process (Though main process). In the 2nd code, the object is created inside a sub-process of Python where everything takes place which should have behaved just like the first one.

Can anybody explain why in the 2nd code, the thread is unable to use the shared memory of the parent object?
I want the output 1 in code 2 i.e. force the thread to use the shared common memory of the parent object. What modification do I need to make?

When you spam the second process, is created about a copy of the first one, so the threads spamed in the 2nd process can only acces that process stack — Netwave, Sep 22 '16 at 08:34
@DanielSanchez Thanks for pointing out the mistake. Changed the position of object creation and it works. Can you add it as an answer so that I can accept? — RatDon, Sep 22 '16 at 08:40
"Kinda sorta" duplicate of http://stackoverflow.com/questions/27809586/unexpected-memory-footprint-differences-when-spawning-python-multiprocessing-poo/29838782#29838782 — temoto, Sep 22 '16 at 10:08
@temoto The question here mentioned is kind of a confusion. Since the whole class is inheriting the multiprocessing, hence the thought was like, whatever being initialized in the constructor would be shared which is not. By the way, thanks. — RatDon, Sep 22 '16 at 10:37

score 1 · Accepted Answer · edited Sep 22 '16 at 10:44

When you spawn the second process, is created about a copy of the first one, so the threads spawned in the 2nd process can only access that process stack.

In code2 The object of myTest is created in the constructor (__init__), so it won't share the same memory as of the spawned process which is run. Just by changing the position of the object creation from constructor to run, the object created becomes part of the spawned process memory.

score 1 · Answer 2 · edited May 23 '17 at 12:16

Forked process sees snapshot of parent's memory at the moment of fork(). When either parent or child changes a single bit, their memory mappings diverge, one sees what it changed, the other sees old state. I've explained the theory more verbose in this answer: https://stackoverflow.com/a/29838782/73957

If you want to share memory [changes], you have to do it implicitly. In Python one of simple ways is multiprocessing.{Queue,Value,Array}, see https://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes

Python multithreading doesn't use common memory when the parent object is created inside a process

2 Answers2