25

This is a follow up to this question, but if I want to pass an argument to stdin to subprocess, how can I get the output in real time? This is what I currently have; I also tried replacing Popen with call from the subprocess module and this just leads to the script hanging.

from subprocess import Popen, PIPE, STDOUT
cmd = 'rsync --rsh=ssh -rv --files-from=- thisdir/ servername:folder/'
p = Popen(cmd.split(), stdout=PIPE, stdin=PIPE, stderr=STDOUT)
subfolders = '\n'.join(['subfolder1','subfolder2'])
output = p.communicate(input=subfolders)[0]
print output

In the former question where I did not have to pass stdin I was suggested to use p.stdout.readline, there there is no room there to pipe anything to stdin.

Addendum: This works for the transfer, but I see the output only at the end and I would like to see the details of the transfer while it's happening.

Community
  • 1
  • 1
hatmatrix
  • 42,883
  • 45
  • 137
  • 231

3 Answers3

39

In order to grab stdout from the subprocess in real time you need to decide exactly what behavior you want; specifically, you need to decide whether you want to deal with the output line-by-line or character-by-character, and whether you want to block while waiting for output or be able to do something else while waiting.

It looks like it will probably suffice for your case to read the output in line-buffered fashion, blocking until each complete line comes in, which means the convenience functions provided by subprocess are good enough:

p = subprocess.Popen(some_cmd, stdout=subprocess.PIPE)
# Grab stdout line by line as it becomes available.  This will loop until 
# p terminates.
while p.poll() is None:
    l = p.stdout.readline() # This blocks until it receives a newline.
    print l
# When the subprocess terminates there might be unconsumed output 
# that still needs to be processed.
print p.stdout.read()

If you need to write to the stdin of the process, just use another pipe:

p = subprocess.Popen(some_cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
# Send input to p.
p.stdin.write("some input\n")
p.stdin.flush()
# Now start grabbing output.
while p.poll() is None:
    l = p.stdout.readline()
    print l
print p.stdout.read()

Pace the other answer, there's no need to indirect through a file in order to pass input to the subprocess.

Alp
  • 2,766
  • 18
  • 13
  • i dunno I couldnt get `sys.stdin.read()` to return anything in the called file with python2.6 with this method ... thats why I didnt put that as an answer ... but maybe i broke something. – Joran Beasley Jul 01 '13 at 21:34
  • 2
    When called with no argument `read()` won't return until it encounters `EOF` (it's trying to read the entirety of the file object it's called on, and it doesn't know it's consumed all of the file until it reaches `EOF`). That's rarely the behavior one wants when reading input. You should either pass a specified number of bytes as argument to `read()` (in which case the `read()` call will return after encountering that number of bytes or `EOF`, whichever comes first) or use `readline()` (which will return after encountering a newline). – Alp Jul 01 '13 at 21:53
  • 2
    your code will deadlock if subprocess generates enough output on stderr. Don't use `stderr=PIPE` unless you read from `p.stderr` later. – jfs Mar 04 '14 at 14:41
  • Removed the unneeded `stderr` kwarg. – Alp Mar 04 '14 at 15:59
  • call `p.stdin.close()` otherwise the child process may not see the input due to buffering. – jfs Mar 05 '14 at 12:41
  • `flush()` would do as well. – Alp Mar 05 '14 at 15:17
  • 1
    When I tried this I got extra whitespace, because the readline() method returns the newline, and print() adds a newline. I changed the print statements to sys.stdout.write() and got the output I expected. – Mark E. Hamilton Oct 17 '14 at 20:51
  • 1
    Is there is a way to read *either* stdout and stderr at the same time? Meaning, like if you would run a shell command, it would print whichever comes first. I'd like to have same behavior.. Since p.stdout.readline() is blocking, not sure how to achieve this. Thanks – Tagar Nov 12 '18 at 05:20
3

something like this I think

from subprocess import Popen, PIPE, STDOUT

p = Popen('c:/python26/python printingTest.py', stdout = PIPE, 
        stderr = PIPE)
for line in iter(p.stdout.readline, ''):
    print line
p.stdout.close()

using an iterator will return live results basically ..

in order to send input to stdin you would need something like

other_input = "some extra input stuff"
with open("to_input.txt","w") as f:
   f.write(other_input)
p = Popen('c:/python26/python printingTest.py < some_input_redirection_thing', 
         stdin = open("to_input.txt"),
         stdout = PIPE, 
         stderr = PIPE)

this would be similar to the linux shell command of

%prompt%> some_file.o < cat to_input.txt

see alps answer for better passing to stdin

Joran Beasley
  • 110,522
  • 12
  • 160
  • 179
  • Thanks, this was an answer given in my previous question where I didn't need `stdin`, but I do not know how to pass to it the 'input' argument as I would to `p.communicate` – hatmatrix Jul 01 '13 at 19:25
  • see edits ... you would need to put the input string in a file and pass an open filehandler to the stdin argument of the constructor – Joran Beasley Jul 01 '13 at 19:57
2

If you pass all your input before starting reading the output and if by "real-time" you mean whenever the subprocess flushes its stdout buffer:

from subprocess import Popen, PIPE, STDOUT

cmd = 'rsync --rsh=ssh -rv --files-from=- thisdir/ servername:folder/'
p = Popen(cmd.split(), stdout=PIPE, stdin=PIPE, stderr=STDOUT, bufsize=1)
subfolders = '\n'.join(['subfolder1','subfolder2'])
p.stdin.write(subfolders)
p.stdin.close() # eof
for line in iter(p.stdout.readline, ''):
    print line, # do something with the output here
p.stdout.close()
rc = p.wait()
jfs
  • 399,953
  • 195
  • 994
  • 1,670