Calling parent instance thru child in subprocess python

Question

I have a file parent.py that calls its child called child.py using subprocess. Similar to this:

p = subprocess.Popen(["C:/Users/.../child.py", some_arg1, some_arg2],shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)

After the child process is started I would like to send a variable from the child process to the parent process. The child should not send a message when it is finished but at a certain point in its execution. How can I achieve this? After searching for several hours I only seem to find a solution the other way around (So from Parent to child). Thanks for your help.

Well, you could either create some sort of communication mechanism using sockets or put a simple cache file with info somewhere which main process listens for and then cleans after it gets the signal. — Bijay Regmi, Apr 16 '22 at 22:02
Does the child process output on both stdout and stderr? If not, you could use that for inter-process communication. Otherwise you'd need to establish communication through a named pipe or similar. — Pi Marillion, Apr 17 '22 at 01:44

score 1 · Accepted Answer · answered Apr 17 '22 at 02:13

There are a few ways to facilitate this sort of inter-process communication. One of the more common ways is with a first-in-first-out (FIFO) named pipe.

Here's a very basic demo.

parent.py:

#! /usr/bin/env python3

import os, tempfile, subprocess

# make a temp directory that's automatically removed when we're done
with tempfile.TemporaryDirectory() as dir:

    # create a FIFO in that directory
    fifo_path = os.path.join(dir, 'fifo')
    os.mkfifo(fifo_path)

    # start the child process
    proc = subprocess.Popen(['./child.py', '--fifo', fifo_path])
    print('process started')

    # open the FIFO
    with open(fifo_path, 'r') as fifo:

        # read output from the child process
        mid_output = fifo.readline()
        print(f'{mid_output = }')

        # wait for child to finish
        code = proc.wait()
        print(f'process finished')

child.py:

#! /usr/bin/env python3

import argparse, time

# read FIFO path from command line
parser = argparse.ArgumentParser()
parser.add_argument('--fifo', required=True)
args = parser.parse_args()

# open FIFO (created by parent)
with open(args.fifo, 'w') as fifo:

    # simulate some work being done
    time.sleep(1)

    # tell the parent that progress has been made
    fifo.write('Halfway there!\n')
    fifo.flush()  # make sure to flush FIFOs

    # Simulate some more work being done
    time.sleep(1)

Then, it runs like so:

./parent.py
process started
mid_output = 'Halfway there!\n'
process finished

You can have the child script output whatever it needs to say to the parent, and you can have it do so multiple times. Just make sure that the parent knows to read from the child the same number of times that the child writes.

@user:2892254 Thanks for your advice. I tried to integrate it within my program but then I noticed that os.mkfifo() isn't working on windows. On StackOverflow I found os.pipe() representing this feature on windows, how could that be integrated? — Gordian, Apr 17 '22 at 09:25
@Dotolox I'm not sure `os.pipe()` will work in this case, since you need a way to pass the pipe through `subprocess`. However, `os.pipe()` *will* work if you use threading instead of a child process. As a side note, child processes are cheaper in Lunix/Mac/Unix than in Windows, and threads can be cheaper in Windows than in Linux/Mac/Unix. — Pi Marillion, Apr 17 '22 at 14:29
Thanks. The https://stackoverflow.com/questions/48542644/python-and-windows-named-pipes could solve this for me on windows — Gordian, Apr 17 '22 at 15:52

Calling parent instance thru child in subprocess python

1 Answers1