Problem
I am writing a piece of software in which I would like to share an object from a certain module. This object should be modifiable from different modules, and within different processes. Consider following (simplified) version of the problem:
Modules
module_shared.py
# Example class with simplified behaviour
class Shared:
def __init__(self):
self.shared = dict()
def set(self, **kwargs):
for key, value in kwargs.items():
self.shared[key] = value
def get(self, *args):
return {key: self.shared[key] for key in args} if args else self.shared
# Module-scope instance of the Shared class
shared = Shared()
module_a.py
from multiprocessing import Process
from time import sleep
import module_shared as ms
def run():
Process(target=run_process).start()
def run_process():
i = 0
while True:
sleep(3)
ms.shared.set(module_a=i)
i+=1
print("Shared from within module_a", ms.shared.get())
module_b.py
from multiprocessing import Process
from time import sleep
import module_shared as ms
def run():
Process(target=run_process).start()
def run_process():
i = 0
while True:
sleep(2)
ms.shared.set(module_b=i)
i-=1
print("Shared from within module_b", ms.shared.get())
module_main.py
import module_a
import module_b
import module_shared as ms
from time import sleep
if __name__ == '__main__':
module_a.run()
module_b.run()
while True:
sleep(5)
print("Shared from within module_main", ms.shared.get())
Output
The output of running module_main is as follows:
Shared from within module_b {'module_b': 0}
Shared from within module_a {'module_a': 0}
Shared from within module_b {'module_b': -1}
Shared from within module_main {}
Shared from within module_a {'module_a': 1}
Shared from within module_b {'module_b': -2}
...
Expected output is as follows:
Shared from within module_b {'module_b': 0}
Shared from within module_a {'module_a': 0, 'module_b': 0}
Shared from within module_b {'module_a': 0, 'module_b': -1}
Shared from within module_main {'module_a': 0, 'module_b': -1}
Shared from within module_a {'module_a': 1, 'module_b': -1}
Shared from within module_b {'module_a': 1, 'module_b': -2}
...
Further explanation
The shared instance is not modified globally because each Process has its own memory space. Initially I have tried fixing it using the Manager from multiprocessing module, however, I have failed to set it up, I presume due to the errors with when and how the import statements are executed. Here is the error message when calling Manager() in Shared's __init__:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
At the moment the best solution was to use threading, however I would prefer to use the processes instead. Naturally, in case any simpler (or better) solutions exist, I would be very happy to consider them.
EDIT:
I have realised I made a typo in my previous attempt with threading, and using multiple threads actually works perfectly fine. Such a great lesson to learn to read your code twice...
