Saving stdout from subprocess.Popen to file line by line

Question

My python script uses subprocess to call an another script, which produces output very slow(line-by-line basis). I would like to write the output line by line to file not when the whole process ends and writes the entire output as string.The following code writes the output to "file" when the "script" ends.

args = ("script")
file = open('output.txt', 'w')
subprocess.Popen(args,stdout=file)

Is it even possible ? Thanx, Chris

Answer 1

You can interact with the process using poll so that you can attempt to interact with it line by line:

For example:

process = subprocess.Popen(["ls", "-lart"],
                 bufsize=-1, # fully buffered (default)
                 stdin=subprocess.PIPE,
                 stdout=subprocess.PIPE,
                 stderr=subprocess.PIPE,
                 cwd=os.curdir,
                 env=os.environ)
my_stdout_file = open("stdout.txt", "w")
while True:
    process.poll()
    line = process.stdout.readline()
    my_stdout_file.write(line)
    eline = process.stderr.readline()
    if line:
        stdout_lines.append(line)
    if eline:
        stderr_lines.append(eline)
    if (line == "" and eline == "" and
        process.returncode != None):
        break

Answer 2

Yes, it is possible. Here is a function that I wrote for a test harness use to do unit testing of Python shell scripts.

def testrun(cmdline):
   try:
      cmdout, cmderr = "",""
      cmdp = Popen(cmdline, shell=True,stdout=PIPE, stderr=PIPE)
      cmdout,cmderr =  cmdp.communicate()
      retcode = cmdp.wait()
      if retcode < 0:
         print >>sys.stderr, "Child was terminated by signal", -retcode
      else:
         return (retcode,cmdout,cmderr)
   except OSError, e:
      return (e,cmdout,cmderr)

The function returns a tuple which contains the shell return code issues by sys.exit(), the standard output text, and the standard error output text. They are both text strings so you would need to use splitlines to break them into lines before processing.

If you really need to interact with the output, line by line, then it is probably better to use pexpect rather than the subprocess module.

Answer 3

Thought I'd share a solution that doesn't use .poll(), .wait() or .communicate(). A couple of points:

• I use import codecs because my output includes East Asian UTF-8 text
• I trap each line with try: to filter out corrupted/invalid UTF-8 text
• I use '\x0a' to force Linux newline regardless of the platform.
• Use for line in iter(subproc.stderr.readline, ''): if you need to capture stderr
• This approach generates output only when child program creates output
• Using the kw dictionary is overkill for this example, but shows how to use **kwargs with subprocess

Code:

import subprocess
import codecs
import os

kw = {
    'bufsize': 0,
    'executable': None,
    'stdin': subprocess.PIPE,
    'stdout': subprocess.PIPE,
    'stderr': subprocess.PIPE,
    'preexec_fn': None,
    'close_fds': False,
    'shell': False,
    'cwd': None,
    'env': None,
    'universal_newlines': False,
    'startupinfo': None,
    'creationflags': 0,
    }

args = ['ls', '-lart']
kw['cwd'] = os.path.expanduser('~')
logfile = os.path.expanduser('~/stdout.txt')
stdlog = []

try:
    subproc = subprocess.Popen(args,**kw)
except:
    print 'Error loading subprocess. Check arguments and kwargs'
    exit()

log = codecs.open(logfile,'w','utf-8')
log.write(': Starting log for: \"%s\"\x0a'%(' '.join(args)))
for line in iter(subproc.stdout.readline, ''):
    try:
        stdlog.append(line.rstrip().decode('utf-8'))
        log.write(stdout[-1]+'\x0a')
        print stdout[-1]
    except:
        pass

log.flush()
log.close()

Answer 4

I had the same problem for a programming language I'm working on, and ended up doing this: https://github.com/perimosocordiae/plumbum/blob/master/lib/stdlib.py#L21

Unfortunately, it involves reading from the output stream a character at a time, accumulating the line until a newline is found. It works, though, and I don't know of any other way to get the same behavior.