subprocess.Popen doesn't redirect output of the spawned process

Question

I am having difficulty redirecting the output of a process created using subprocess.Popen to a file of my choice. The following is what I am doing:

#popen_test.py 
import sys, os, subprocess, time, signal

process_log = open('process.log','w')
process = subprocess.Popen([<path_to_python_binary>,"process.py"], stdout=process_log, stderr=process_log, preexec_fn=os.setsid) #launching the sub process
time.sleep(10) #waiting for sometime; this allows the sub process to print some statements. See below
process_log.flush()
os.killpg(process.pid,signal.SIGTERM)
process_log.close()

And, this is how my process.py looks. It is basically printing some statements on stdout at regular intervals:

#process.py    
import sys,os,time
print "Initially..."
while True:
    print "Hello World"
    time.sleep(1)

The process is being launched as I can see using ps, but the output is not going to the process.log--the file remains empty. Any idea what I might be doing wrong?

Try using a full-path to the file when you open it in `process_log = open('process.log','w')` — Nir Alfasi, Feb 22 '15 at 04:19
unlrelated: you could use `p.terminate()` without `setsid` hack here because `process.py` does not create its own child processes. — jfs, Feb 22 '15 at 04:54

score 3 · Accepted Answer · edited May 23 '17 at 11:58

It looks like a "block-buffering mode"-related issue. Run the script using python -u or add sys.stdout.flush() after print "hello world".

sys.stdout is fully buffered when redirected to a file by default. print "hello world" adds the string to the stdout buffer. The buffer is not flushed if you kill the subprocess and therefore nothing is written to the file if the buffer does not overflow while the process is still running (buffer size 4K-8K: os.fstat(sys.stdout.fileno()).st_blksize).

Note: the child process works with a copy of the file descriptor i.e., you can close process_log file in the parent as soon as Popen() returns -- it has no effect on the child.

subprocess.Popen doesn't redirect output of the spawned process

1 Answers1