How to wait for the start of a child process?

Question

I execute a process (using Popen) and after a while, this process executes a child process.
What I'm trying to do is wait for the child process to be spawned and then continue executing my script.

The only way that I found, is using psutil and polling until the process has children.

process = psutil.Process(subprocess.Popen(command).pid)
while 0 == len(process.children()):
    time.sleep(0.2)

Is there a better way of doing it than polling on the children? Maybe waiting for some event?

I'm using windows 10 for this.

Is is an option that the first child writes something on its standard output or output error after spawning its own child? Or use any other way to signal the operation.? — Serge Ballesta, Jul 26 '17 at 12:59
no any another way. no any special events when one process create another (only kernel mode callback called) — RbMm, Jul 26 '17 at 13:00
@SergeBallesta It's possible for the process to do something after spawning the child process. (Like printing something to the standard output) — cydan, Jul 26 '17 at 13:02
@RbMm, but there is an event for adding a process to a Job. The child process could be created suspended and added to a Job that doesn't allow breakaway and which is associated with a Completion Port. Then when the child is resumed and in turn spawns another process, the Job will post a `JOB_OBJECT_MSG_NEW_PROCESS` to the Completion Port, which will contain the PID of the grandchild process. The grandparent process waits for this message via `GetQueuedCompletionStatus`. — Eryk Sun, Jul 26 '17 at 14:18
@eryksun - i mean in general case, if specific process special design for notify when he create child - another situation — RbMm, Jul 26 '17 at 14:37
@RbMm, this requires no modification to either the child or grandchild process. It's all handled by the Windows Job object. — Eryk Sun, Jul 26 '17 at 14:39
@eryksun - yes, you right. in almost all case this will be good solution. if child not try create process detached from job — RbMm, Jul 26 '17 at 15:17

score 1 · Answer 1 · answered Jul 26 '17 at 13:45

Based on your comment that you can have the process you're lunching write something to STDOUT when it launches its subprocess, you can easily use Popen.check_output() to pick up the STDOUT string from your parent process. For example, if your subprocess (the command) writes to STDOUT Subprocess started\n, you can do it purely through the subprocess module as:

import subprocess

response = subprocess.check_output(command)
if response.rstrip() == "Subprocess started":
    print("Woo! Our sub-subprocess was started...")

However, if your command returns multiple outputs, you might have to weed out the ones you're not interested in. You can do that by observing your subprocess' STDOUT for the aforementioned Subprocess started string, something like:

import subprocess

proc = subprocess.Popen(command, stdout=subprocess.PIPE)
while True:  # a loop to read our subprocess STDOUT
    line = proc.stdout.readline().rstrip()  # read line-by-line
    if line == "Subprocess started":
        print("Woo! Our sub-subprocess was started...")
        break  # no need to read the STDOUT anymore

You can also capture STDERR (or pipe it to STDOUT) if you expect to be receiving error messages from your command (e.g. cannot start the subprocess or something like that).

The problem is, that I still have to wait for the output so I still need to poll using time.sleep — cydan, Jul 26 '17 at 13:55
@cydan - you don't need to do anything, until your subprocesses STDOUT line-buffer is filled (so at least one line) all of this will wait for it (`proc.stdout.readline()` is a blocking action). `subprocess.check_output()` is also blocking if your subprocess is expected to write something to its STDOUT only once. — zwer, Jul 26 '17 at 14:00
Rather than overloading standard I/O for this task, you could create an inheritable pipe and pass the handle value as a command-line argument or environment variable. If it's also a Python process, the child can associate the handle as a file descriptor via `msvcrt.open_osfhandle`, after which it can be used with regular Python I/O functions. Or just use the handle directly via WinAPI `WriteFile`. — Eryk Sun, Jul 26 '17 at 14:33

score 0 · Answer 2 · answered Jul 26 '17 at 13:29

I think there is no better solution for this. But you could optimize the process call:

process = psutil.Popen(command)

Popen class

A more convenient interface to stdlib subprocess.Popen. It starts a sub process and you deal with it exactly as when using subprocess.Popen but in addition it also provides all the methods of psutil.Process class. [...]

Additionally, you could use write your own event with a wait function. Wait is more CPU hungry but it is more responsive. See here for example. But it depends on your python version.

score 0 · Answer 3 · answered Jul 26 '17 at 13:47

If the first child can write something after spawning its own child, the master script could wait for the first write.

Here is a working example (environment Python 2.7 on Windows):

Python master script:

python = r"C:\Python27\python.exe"
script = r"...\foo.py"
deb = time.clock()
child = subprocess.Popen([ python, script], stdout = subprocess.PIPE)
cr = child.stdout.read()
fin = time.clock()
print fin-deb, cr

Python (simplified) first child:

time.sleep(2)  # add a delay
# the child is supposed to be spawned here
print "Done"
sys.stdout.flush()  # flushes stdout so that caller gets it immediately
time.sleep(8)       # just to make the child live a little longer

The master script output is:

2.12153148707 Done

proving that the master script continued after the first sleep.

score 0 · Answer 4 · answered Jul 27 '17 at 00:41

If you have control over the child process you can use some synchronization primitive between parent and child, like using a socket or a file, otherwise polling over children() in a busy loop is the only thing you can do, just make sure to exit the while loop after a while or your program may end up hanging indefinitely.

How to wait for the start of a child process?

4 Answers4