Getting output of a process at runtime

Question

I am using a python script to run a process using subprocess.Popen and simultaneously store the output in a text file as well as print it on the console. This is my code:

result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for line in result.stdout.readlines(): #read and store result in log file
    openfile.write("%s\n" %line)
    print("%s" %line)

Above code works fine, but what it does is it first completes the process and stores the output in result variable. After that for loop stores the output as well as print it.

But i want the output at runtime (as my process can take hours to complete, i don't get any output for all these hours).

So is there any other function that gives me the output dynamically (at runtime), means as soon as the process gives first line, it should get printed.

I've never actually tried it, but I think you are supposed to send your own Python `file` object to the `stdout` (or `stdin`, or `stderr`) argument. Then you have to poll that file. `Subprocess` was invented to spare you that pain, but it looks like you have no choice. Good luck. — Adrian Ratnapala, May 15 '13 at 06:13
_"it first completes the process and stores the output in result"_ - that's just not true. — Eric, May 15 '13 at 06:50
"it first completes the process and stores the output in result" - that's just not true. Well, when i am running Popen command, it keeps on running until the process finishes and then it executes further coding. If its work some other way, i am eager to know that. — Niyojan, May 15 '13 at 08:13
@AdrianRatnapala Sounds too complicated, and the "no choice" claim is wrong. — glglgl, May 15 '13 at 09:38
you may consider using fcntl to set `result.stdout` as non blocking, this will allow to read in real time and if there is no data yet, `IOError` will be raised — EmilioPeJu, Mar 13 '18 at 12:04

score 8 · Accepted Answer · edited Jul 08 '22 at 16:28

8

The problem here is that .readlines() gets the entire output before returning, as it constructs a full list. Just iterate directly:

for line in result.stdout:
    print(line)

edited Jul 08 '22 at 16:28

Copas

5,921
5
29
43

answered May 15 '13 at 06:52

Eric

95,302
53
242
374

1

note: [`for line in result.stdout` does not return lines in "real time" on Python 2, use `for line in iter(result.stdout.readline, b'')` instead](http://ideone.com/8KFawl) (your `print line` statement suggests that you expect it to work on Python 2. [On Python 3 it works fine](http://ideone.com/EUpeGV) – jfs May 16 '13 at 01:59
Yes, its working, i am getting output at real time, Thank you. – Niyojan May 16 '13 at 04:50

score 4 · Answer 2 · edited May 23 '17 at 12:02

.readlines() returns a list of all the lines the process will return while open, i.e., it doesn't return anything until all output from the subprocess is received. To read line by line in "real time":

import sys
from subprocess import Popen, PIPE

proc = Popen(cmd, shell=True, bufsize=1, stdout=PIPE)
for line in proc.stdout:
    openfile.write(line)
    sys.stdout.buffer.write(line)
    sys.stdout.buffer.flush()
proc.stdout.close()
proc.wait()

Note: if the subprocess uses block-buffering when it is run in non-interactive mode; you might need pexpect, pty modules or stdbuf, unbuffer, script commands.

Note: on Python 2, you might also need to use iter(), to get "real time" output:

for line in iter(proc.stdout.readline, ""):
    openfile.write(line)
    print line,

score 1 · Answer 3 · answered May 15 '13 at 06:36

1

You can iterate over the lines one by one by using readline on the pipe:

while True: 
    line = result.stdout.readline()
    print line.strip()
    if not line:
        break

The lines contain a trailing \n which I stripped for printing. When the process terminates, readline returns an empty string, so you know when to stop.

answered May 15 '13 at 06:36

Thomas Fenzl

4,342
1
17
25

1

I am sorry but you didn't get my question. What you have suggested i am already accomplishing it using for loop, what i want is line by line output while process is still in running state. – Niyojan May 15 '13 at 07:55
1

did you try? You're using `readlines`, which returns all lines. That's only possible after termination. With `readline`, you get only the next – Thomas Fenzl May 15 '13 at 08:31
result.stdout works just fine for me, but your line.strip() functions is helpful as i was getting too many "b'\r\n'" before it, Thank you – Niyojan May 16 '13 at 04:51

Getting output of a process at runtime

3 Answers3

Linked