Child process call function in parent process (python multiprocessing)?

Question

How can I have a child process call a function in its parent process?

I'm writing a python program in which I need a child process (launched with the multiprocessing module) to call a function in its parent process.

Consider the following program that simply calls a function child_or_parent() twice: first it calls child_or_parent() in the parent process, and then it calls child_or_parent() in a child process.

#!/usr/bin/env python3
import multiprocessing, os

# store the pid of our main (parent) process
parent_pid = os.getpid()

# simple function that tells you if it's the parent process or a child process
def child_or_parent():
    if os.getpid() == parent_pid:
        print( "I am the parent process" )
    else:
        print( "I am a child process" )

# first the parent process
child_or_parent() 

# now a child process
child = multiprocessing.Process( target=child_or_parent )
child.start()

When executed, the above program outputs the following

I am the parent process
I am a child process

I would like to modify this program such that the child process calls the parent processes' child_or_parent() function, and it outputs I am the parent process.

The following program has a function named child_call_parent() that contains pseudocode. What should I put there so that the child_call_parent() function will actually execute the child_or_parent() function inside the parent process?

#!/usr/bin/env python3
import multiprocessing, os

# store the pid of our main (parent) process
parent_pid = os.getpid()

# simple function that tells you if it's the parent process or a child process
def child_or_parent():
    if os.getpid() == parent_pid:
        print( "I am the parent process" )
    else:
        print( "I am a child process" )

def child_call_parent():

    # FIXME this does not work!
    self.parent.child_or_parent()

# have a child call a function in its parent process
child = multiprocessing.Process( target=child_call_parent )
child.start()

what's the goal exactly in calling the parent function? You can share data or transmit data between parent and child, but a function is a function, whether you call it from child or parent, the difference is the context to that function, so if you're talking about sharing context, then you should use the messaging functionality like queues. — MrE, Jun 16 '23 at 03:12
sometimes you need to execute a method of an object asynchronously in the background in a child process, but the actions of that child need to act on data that cannot be pickled, so the child needs to communicate back up to the parent to interact with the unpickleable variable. But this is really out-of-scope of this simplified question. — Michael Altfield, Jun 16 '23 at 03:14

Michael Altfield · Answer 1 · 2023-06-16T03:12:24.550

0

I don't know if this is possible directly -- BUT you can use a Queue to pass the name of the function-to-be-called from the child up to the parent, and have the parent just poll the queue and call the function that it's given by the child when it pulls the name of the function out of the queue.

#!/usr/bin/env python3
import multiprocessing, os

# store the pid of our main (parent) process
parent_pid = os.getpid()

# simple function that tells you if it's the parent process or a child process
def child_or_parent():
    if os.getpid() == parent_pid:
        print( "I am the parent process" )
    else:
        print( "I am a child process" )

def child_call_parent(q):
    q.put( "child_or_parent" )

# have a child call a function in its parent process
q = multiprocessing.Queue()
child = multiprocessing.Process( target=child_call_parent, args=(q,) )
child.start()

while True:
    if not q.empty():
        function_name = q.get()

        # call the function given to us
        #  * https://stackoverflow.com/questions/3061/calling-a-function-of-a-module-by-using-its-name-a-string
        function = locals()[ function_name ]
        function()
        break

Example execution

I am the parent process

The `while True:` and subsequent call to `q.empty()` within the block are not necessary since you are doing a blocking `get` call on the queue. Besides, the call to `empty` is not even reliable (see the documentation). But even if it were reliable, you are needlessly burning up CPU cycles by repeatedly making calls to `empty` until a message arrives. The whole block *could* even be replaced with just `locals()[q.get()]()`. – Booboo Jun 16 '23 at 18:09
The `q.get()` call here **is not blocking** because it's encased in the `q.empty()` if statement. Not sure why `q.empty()` would burn up many CPU cycles, but I do have a `time.sleep(0.01)` in [my actual implementation](https://github.com/BusKill/buskill-app/blob/8715cbab257b0d4d858a8cfa4cb179e953d1f303/src/packages/buskill/__init__.py#L998-L1017) of this. – Michael Altfield Jun 16 '23 at 18:59
If I understand what you are trying to do, you just want to wait until there is a message on the queue and then get it. The simplest and most efficient way of doing that is to use a blocking `get` call, which you are in fact using. So far so good. But therefore, the testing of whether there is a message on the queue with the call `q.empty()` is not only unnecessary, it is a call that will not reliably tell you whether there is something on the queue. Additionally, looping on this call is also wasteful. And as far calling `sleep`, I can only go by what you post. (more...) – Booboo Jun 16 '23 at 19:26
But by sleeping, you are almost guaranteed *not* to get any message on the queue as soon as it is actually "get-able". – Booboo Jun 16 '23 at 19:28
Also, you are passing via the queue a function name. But what if that function had arguments? You are not in any way passing to the main process what those arguments might be. So this is not a very generalized capability allowing you to call any *arbitrary* function in the main process. Have you looked at my answer? I have used a `Pipe` instead of a `Queue`, which for this particular application is sufficient and has less overhead than a `Queue` (which, by the way, is built using a `Pipe`), but the idea is the same. I can call *any* arbitrary function and get back a return value. – Booboo Jun 16 '23 at 19:39

score 0 · Answer 2 · answered Jun 16 '23 at 17:53

In the code below I have created a multiprocessing.Pipe with its two connection objects (simpler than using a Queue) and the connections are passed to the child process. The child process can send through conn2 a request to the main process to execute an arbitrary function call and return the results. Because the function call has arguments, we use function functools.partial to compose a modified function call that incorporates the required arguments.

I have also modified the code slightly to work on all platforms. The original code had the statement parent_pid = os.getpid() at global scope. On Windows, this would be re-executed in the child process and therefore erroneously assign the child process's PID to parent_pid. Now the parent_id value is created only in the main process and then explicitly passed as an argument to whomever needs it.

#!/usr/bin/env python3
import multiprocessing
import os
from functools import partial

# simple function that tells you if it's the parent process or a child process
def child_or_parent(parent_pid):
    if os.getpid() == parent_pid:
        print("I am the parent process")
    else:
        print("I am a child process")

def call_child_or_parent(parent_id, conn1, conn2):
    fn = partial(child_or_parent, parent_id)
    # child_or_parent(parent_id) is invoked from this child process:
    fn()
    # Have parent call child_or_parent(parent_id):
    conn2.send(fn)
    results = conn1.recv()

# Modified to run also on platforms that create child processes
# using the spawn method:
if __name__ == '__main__':
    parent_pid = os.getpid()

    conn1, conn2 = multiprocessing.Pipe(duplex=False)
    child = multiprocessing.Process(target=call_child_or_parent, args=(parent_pid, conn1, conn2))
    child.start()

    # Get function call request and return results:
    fn = conn1.recv()
    conn2.send(fn())

    child.join()

Prints:

I am a child process
I am the parent process

Child process call function in parent process (python multiprocessing)?

2 Answers2

Example execution

Further Reading