How can I update class members in processes?

Question

I have looked for other questions, and this un-accepted-answered question is the only one I could find that somehow covers this issue and is not really helpful. Also, I need this to work with processes, and not threads.

So from the ground up I wrote a sample program to show my issue, you should be able to paste it and it will run:

import multiprocessing
import time 

class Apple:
   def __init__(self, color):
      self.color = color

def thinkAboutApple(apple):
   while True:
      print(apple.color)
      time.sleep(2)

my_apple = Apple("red")
new_process = multiprocessing.Process(target=thinkAboutApple, args=(my_apple,))
new_process.start()
time.sleep(4)
print("new: brown")
my_apple.color = "brown"

#so that the program doesn't exit after time.sleep(4)
while True:
    pass

# actual output | # wanted output
red             | red
red             | red
new: brown      | new: brown
red             | brown
red             | brown

This tells me that either the apple is in a weird supposition where it is two colours at the same time, OR that the new_process' apple is in another position in ram and separated from the apple in the main process.

So the question is: Is there a way to have the pointer of the apple in the process point to the same apple, or what is the pythonic way to keep all instances of the apple in all processes the same? What if I have the same apple in many processes and even more processes without the apple, how do I make sure they area always the same?

Use threads instead of processes. Processes don’t share data unlike threads. If you want to use processes, you would need to provision shared memory constructs. See [this](https://docs.python.org/3/library/multiprocessing.shared_memory.html). — Jarvis, Dec 25 '20 at 20:20
@Jarvis It's not clear from this minimal example if replacing processes with threads is feasible. — chepner, Dec 25 '20 at 20:23
@Jarvis thanks for the suggestion, however for this situation I have deliberately chosen processes over threads. I know one of the disadvantages is the complicated communication, but I need the parrallelism — Maritn Ge, Dec 25 '20 at 20:27
@chepner yea I don't think I could create an example which is easy enough for a SO question to make it clear that I need processes, so I added that info to the question — Maritn Ge, Dec 25 '20 at 20:29
Checkout [this](https://stackoverflow.com/a/39742800/48063570) — Jarvis, Dec 25 '20 at 20:32
@Jarvis I think this is a python-2 only solution, as the print has no (), and pasting the answer into an empty .py doesn't run because 'instance of Basemanager has no Simpleclass' — Maritn Ge, Dec 25 '20 at 20:46
@MaritnGe - Along with the obtained output, could you please also mention the expected output? — fountainhead, Dec 25 '20 at 20:52
Are you trying to drive a GUI, or get a deterministic output from a state engine? You need either some kind of messaging like https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.connection or you need to get rid of the "parallelism" entirely. I can get the expected output with a complete redesign and no multiprocessing, but I expect that doesn't meet your need. — Kenny Ostrom, Dec 25 '20 at 22:07
@KennyOstrom I fear I will have to use a lot of sockets, pipe and tears to get this working, i think maybe I'll for now try it with threads to move along and keep looking for a processing solution — Maritn Ge, Dec 25 '20 at 22:10
Look up the game of life for a state engine approach. Then you just need the main thread to sleep between states. One key point -- I don't think you want different processing printing. Only one process should be allowed to do IO. — Kenny Ostrom, Dec 25 '20 at 22:11

martineau · Accepted Answer · 2022-02-10T21:48:17.660

You can derive a specialized version of a Proxy class used by multiprocessing.BaseManager from the (undocumented) multiprocessing.managers.NamespaceProxy class that, unlike the base class, exposes all of its methods and attributes. This is similar to @shtse8's answer to the linked duplicate question, but I'm posting a runnable answer to it here to make clear how it can be done.

from multiprocessing import Process
from multiprocessing.managers import BaseManager, NamespaceProxy
import time
import types

class MyManager(BaseManager): pass  # Avoid namespace pollution.

class Apple:
    def __init__(self, color):
        self.color = color


def Proxy(target):
    """ Create a derived NamespaceProxy class for `target`. """
    def __getattr__(self, key):
        result = self._callmethod('__getattribute__', (key,))
        if isinstance(result, types.MethodType):
            def wrapper(*args, **kwargs):
                self._callmethod(key, args)
            return wrapper
        return result

    dic = {'types': types, '__getattr__': __getattr__}
    proxy_name = target.__name__ + "Proxy"
    ProxyType = type(proxy_name, (NamespaceProxy,), dic)  # Create subclass.
    ProxyType._exposed_ = tuple(dir(target))

    return ProxyType


AppleProxy = Proxy(Apple)


def thinkAboutApple(apple):
    while True:
        print(f"apple.color: {apple.color}")
        time.sleep(1)


if __name__ == '__main__':

    MyManager.register('Apple', Apple, AppleProxy)

    manager = MyManager()
    manager.start()

    my_apple = manager.Apple("red")
    new_process = Process(target=thinkAboutApple, args=(my_apple,))
    new_process.start()

    time.sleep(2)  # Allow other process to run a short while.
    my_apple.color = "brown"  # Change shared class instance.

    time.sleep(2)  # Allow other process to run at little while longer.
    new_process.terminate()

One slight edit: The content of the wrapper function should be return self._callmethod(key, args, kwargs). Otherwise, this swallows and ignores any keyword arguments. And it always returns None. Otherwise, this answer is exactly what I needed. — Mike Jarvis, Dec 03 '22 at 20:46

How can I update class members in processes?

1 Answers1

Linked