11

I found a number of questions which looks like mine, but which did not produce a solution I can use (closest is: subprocess output to stdout and to PIPE)

The problem: I want to start a process using subprocess which takes a long time. After running the command I need to parse the stdout-output and the stderr-output.

Currently I do it as follows:

p = subprocess.Popen( command_list, stdout=subprocess.PIPE, 
    stderr=subprocess.PIPE )
out, error_msg = p.communicate()
print out + "\n\n" + error_msg

#next comes code in which I check out and error_msg

But the drawback of this method is that the user does not see the output of the process while it is running. Only at the end the output is printed.

Is there a way that the output is printed while the command is running (as if I gave the command without stdout/stderr=subprocess.PIPE) and still have the output via p.communicate in the end?

Note: I'm currently developing on python 2.5 (old software release which uses this python version).

Community
  • 1
  • 1
Nemelis
  • 4,858
  • 2
  • 17
  • 37
  • related: [Subprocess.Popen: cloning stdout and stderr both to terminal and variables](http://stackoverflow.com/q/17190221/4279) – jfs Apr 10 '15 at 10:49
  • related: [Displaying subprocess output to stdout and redirecting it](http://stackoverflow.com/q/25750468/4279) – jfs Apr 10 '15 at 10:49
  • related: [Python subprocess get children's output to file and terminal?](http://stackoverflow.com/q/4984428/4279) – jfs Apr 10 '15 at 10:51

1 Answers1

6

This snippet has helped me once in a similar situation:

process = subprocess.Popen(cmd, bufsize=1, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(process.stdout.readline, ''):
    print line,
    sys.stdout.flush() # please see comments regarding the necessity of this line 
process.wait()
errcode = process.returncode
Scis
  • 2,934
  • 3
  • 23
  • 37
  • The `flush()` may be useful if you really require instantaneous output, but it's not strictly necessary if all you care about is to see some progress from a process which generates enough output for buffering to not matter a lot. If it's a very slow writer, of course, it will take a long time before the buffer fills up; and then maybe you do want to `flush()` often. – tripleee Apr 10 '15 at 10:43
  • @tripleee: I find that `sys.stdout.flush()` is unnecessary here if we assume that `python` process and the child process have the same buffering policy (likely if they both stdio-based) i.e., if `python` script is run in the terminal (no redirection) then the stdout is already line-buffered -- no `sys.stdout.flush()` necessary. If the output is redirected to a file (despite what OP asks) then `flush()` unnecessarily slows down the printing (block-buffering that is typically used when the output is file is more efficient, the flushing after each line makes it slower). – jfs Apr 10 '15 at 10:57
  • I would call `flush()` explicitly iff `--line-buffering` option is given (like `grep`). – jfs Apr 10 '15 at 11:01
  • Note: `stderr=STDOUT` merges stdout/stderr here. OP might want to capture them separately. – jfs Apr 10 '15 at 11:02
  • @J.F.Sebastian and tripleee thanks for the extra info I've edited the answer to draw attention to those comments – Scis Apr 10 '15 at 11:14
  • Kudos. That almost solves my problem. As J.F. Sebastian is already stating I need to capture stdout and stderr seperately (and print them at the same time). But this helps – Nemelis Apr 10 '15 at 11:16
  • 1
    @Nemelis: unfortunately, capturing stdout/stderr separately complicates the code a lot: you need threads or `select()` or other asynchronous I/O tools. See the questions that I've linked above. – jfs Apr 10 '15 at 11:23
  • @J.F.Sebastian: Thanx. Will check them out. Already found out that capturing them separately was not easy. – Nemelis Apr 10 '15 at 11:31