How do I get all of the output from my .exe using subprocess and Popen?

Question

I am trying to run an executable and capture its output using subprocess.Popen; however, I don't seem to be getting all of the output.

import subprocess as s
from subprocess import Popen 
import os

ps = Popen(r'C:\Tools\Dvb_pid_3_0.exe', stdin = s.PIPE,stdout = s.PIPE)
print 'pOpen done..'

while:

line = ps.stdout.readline()
print line

It prints two line less than the original exe file when opened manually.

I tried an alternative approach with the same result:

f = open('myprogram_output.txt','w')
proc = Popen('C:\Tools\Dvb_pid_3_0.exe ', stdout =f)

line = proc.stdout.readline()
print line
f.close()

Can anyone please help me to get the full data of the exe?

As asked by Sebastian:

Original exe file last few lines o/p:

-Gdd : Generic count (1 - 1000)

-Cdd : Cut start at (0 - 99) -Edd : Cut end at (1 - 100)

Please select the stream file number below:

1 - .\pdsx100-bcm7230-squashfs-sdk0.0.0.38-0.2.6.0-prod.sao.ts

The o/p I get after running:

-P0xYYYY      : Pid been interested                                          

-S0xYYYY : Service ID been interested
-T0xYYYY : Transport ID been interested
-N0xYYYY : Network ID been interested
-R0xYYYY : A old Pid been replaced by this PID
-Gdd : Generic count (1 - 1000)

So we can see some lines missing. I have to write 1 and choose value after please select the fule number below appears.

I tried to use ps.stdin.write('1\n'). It didn't print the value in the exe file

New code:

#!/usr/bin/env python
from subprocess import Popen, PIPE

cmd = r'C:\Tools\Dvb_pid_3_0.exe'
p = Popen(cmd, stdin=PIPE, stdout=None, stderr=None, universal_newlines=True)
stdout_text, stderr_text = p.communicate(input="1\n\n")

print("stdout: %r\nstderr: %r" % (stdout_text, stderr_text))
if p.returncode != 0:
    raise RuntimeError("%r failed, status code %d" % (cmd, p.returncode))

Thanks Sebastien. I am able to see the entire output but not able to feed in any input with the current code.

Answer 1

To get all stdout as a string:

from subprocess import check_output as qx

cmd = r'C:\Tools\Dvb_pid_3_0.exe'
output = qx(cmd)

To get both stdout and stderr as a single string:

from subprocess import STDOUT

output = qx(cmd, stderr=STDOUT)

To get all lines as a list:

lines = output.splitlines()

To get lines as they are being printed by the subprocess:

from subprocess import Popen, PIPE

p = Popen(cmd, stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, ''):
    print line,
p.stdout.close()
if p.wait() != 0:
   raise RuntimeError("%r failed, exit status: %d" % (cmd, p.returncode))

Add stderr=STDOUT to the Popen() call to merge stdout/stderr.

Note: if cmd uses block-buffering in the non-interactive mode then lines won't appear until the buffer flushes. winpexpect module might be able to get the output sooner.

To save the output to a file:

import subprocess

with open('output.txt', 'wb') as f:
    subprocess.check_call(cmd, stdout=f)

# to read line by line
with open('output.txt') as f:
    for line in f:
        print line,

If cmd always requires input even an empty one; set stdin:

import os

with open(os.devnull, 'rb') as DEVNULL:
    output = qx(cmd, stdin=DEVNULL) # use subprocess.DEVNULL on Python 3.3+

You could combine these solutions e.g., to merge stdout/stderr, and to save the output to a file, and to provide an empty input:

import os
from subprocess import STDOUT, check_call as x

with open(os.devnull, 'rb') as DEVNULL, open('output.txt', 'wb') as f:
    x(cmd, stdin=DEVNULL, stdout=f, stderr=STDOUT)

To provide all input as a single string you could use .communicate() method:

#!/usr/bin/env python
from subprocess import Popen, PIPE

cmd = ["python", "test.py"]
p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines=True)
stdout_text, stderr_text = p.communicate(input="1\n\n")

print("stdout: %r\nstderr: %r" % (stdout_text, stderr_text))
if p.returncode != 0:
    raise RuntimeError("%r failed, status code %d" % (cmd, p.returncode))

where test.py:

print raw_input('abc')[::-1]
raw_input('press enter to exit')

If your interaction with the program is more like a conversation than you might need winpexpect module. Here's an example from pexpect docs:

# This connects to the openbsd ftp site and
# downloads the recursive directory listing.
from winpexpect import winspawn as spawn

child = spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('noah@example.com')
child.expect ('ftp> ')
child.sendline ('cd pub')
child.expect('ftp> ')
child.sendline ('get ls-lR.gz')
child.expect('ftp> ')
child.sendline ('bye')

To send special keys such as F3, F10 on Windows you might need SendKeys module or its pure Python implementation SendKeys-ctypes. Something like:

from SendKeys import SendKeys

SendKeys(r"""
    {LWIN}
    {PAUSE .25}
    r
    C:\Tools\Dvb_pid_3_0.exe{ENTER}
    {PAUSE 1}
    1{ENTER}
    {PAUSE 1}
    2{ENTER}
    {PAUSE 1}
    {F3}
    {PAUSE 1}
    {F10}
""")

It doesn't capture output.

Answer 2

The indentation of your question threw me off a bit, since Python is particular about that. Have you tried something as so:

import subprocess as s
from subprocess import Popen 
import os

ps = Popen(r'C:\Tools\Dvb_pid_3_0.exe', stdin = s.PIPE,stdout = s.PIPE)
print 'pOpen done..'

(stdout, stderr) = ps.communicate()
print stdout

I think that stdout will be one single string of whatever you return from your command, so this may not be what you desire, since readline() presumes you want to view output line by line.

Would suggest poking around http://docs.python.org/library/subprocess.html for some uses that match what you are up to.