Python Process hangs until child process terminate using tee scheme

Question

I have a following problem related to process synchronization.

There is a python script startup.py, an executable maestro, and an executable tee. My python script startup.py starts maestro program and pipes the stderr/stdout of maestro into the log file using tee as well write into the console.

I achieved this using following code:

cmd = "maestro"
mae_err_log = "output.txt"
maestro = subprocess.Popen(cmd, stderr = subprocess.STDOUT, stdout=subprocess.PIPE)
tee = subprocess.Popen(['tee', mae_err_log], stdin = maestro.stdout)
maestro.stdout.close()
tee.communicate()
maestro_status = maestro.returncode
sys.exit(maestro_status)

My maestro program is a gui program when I exit from the maestro program, it internally call posix system("maestro_cleanup &") api and immediately exit. I noticed that my maestro program does not release the console until maestro_cleanup program terminate although I am running maestro_cleanup in the background. I need to run maestro_cleanup in the background as it takes time and I do not want to hold the console for the period until maestro_cleanup finishes. Using above code, maestro program does not terminate until maestro_cleanup finishes.

I see "ps -elf" shows following:

0 S j 16876  6678  0  75 0 - 65307 wait 18:56 pts/53 00:00:00 python startup.py
0 Z j 17230 16876  4  76 0 - 0     exit 18:56 pts/53 00:00:04 [maestro] <defunct>
0 S j 17231 16876  0  77 0 - 948 pipe_w 18:56 pts/53 00:00:00 /usr/bin/tee output.txt
0 S j 17424     1  0  77   0 -   948 -  18:57 pts/53   00:00:00 maestro_cleanup

You can see the maestro_cleanup parent process is session process id instead maestro because I started maestro_cleanup using system api in background.

Any idea why maestro does not terminate until maestro_cleanup finishes?

Any help would be appreciated.

-Vipin

Answer 1

I'd say use close_fd=True. It should do the trick.

Answer 2

Well the subprocess.Popen documentation does say tee.communicate() will "Wait for process to terminate"; also your invocation is the same as this guy who does say his child process runs in the background, so that seems ok. So this blocking-before-exit sounds like a limitation of subprocess.

Try the links in 'Python spawn off a child subprocess, detach, and exit' which references the ActiveState recipe: 'Creating a daemon the Python way': The Python way to detach a process from the controlling terminal and run it in the background as a daemon.