python subprocess output disappear when redirect stdout to pipe

Question

I'm currently working with piece of code that uses a script that gets real time data and print it onto my screen.

When I run the script (in which I don't have the access to view) in python console, using:

>>>> myproc = subprocess.Popen('command')
...print....

The output is perfectly printed onto my screen when the data come.

However, if I change it to

>>>> myproc = subprocess.Popen('command', stdout = subprocess.PIPE, stderr = subprocess.PIPE)`
>>>> while True:
>>>>    print(myproc.stdout.readline())
... perfectly no print....

there is zero output (I have also checked the stderr, nothing)

I guess the problem here is that i set stdout to subprocess.PIPE, and I should probably change that.

However, since i need to store the data that come from myproc, and using stdout is so far the only method that i know that allows me to do that, I'm running out of ideas on how to change my code.

Please tell me what can I do here, and also please tell me why did this happen.

Thanks

EDIT: My script listens to real time data, so it will keep on running until I manually interrupt it. I want prints before it actually terminates.

2nd EDIT :

I found out where the missing out put went... I have to change it to

myproc = subprocess.Popen('command', stdout=sys.stdout) and then use

While True:
    if myproc.stdout is not None:
          print(myproc.stdout)

for the output to show. But this is just super ugly and the output can be extremely unpredictable.. Can anyone maybe provide a better approach to this ugly thing??

What is `myproc.poll()` inside the loop (the process might have failed to start silently)? don't use `stderr=PIPE` unless you read from it (concurrently). Where do you want to save the output data (your code just prints to stdout), do you want to filter data e.g., `sys.stdout.write(your_python_filter(myproc.stdout.readline()))` or just save it to a file or both? Does `command |& cat` work in bash (do you see any output)? Is it a buffering issue? See [Python subprocess readlines() hangs](http://stackoverflow.com/q/12419198/4279) — jfs, May 21 '14 at 16:04
@J.F.Sebastian .poll() will hang because the program doesn't terminate itself — user1948847, May 21 '14 at 16:51
poll() returns immediately. it is not wait(). It returns None if the process is still running. You should break the loop if it returns anything else -- meaning that the child process ended prematurely. — jfs, May 22 '14 at 20:15

zmo · Answer 1 · 2014-05-20T21:55:06.213

You should use the communicate to wait for the process to finish and get the stdout and stderr handlers:

myproc = subprocess.Popen(['command'], stdout = subprocess.PIPE, stderr = subprocess.PIPE)
out, err = myproc.communicate()
print("STDOUT:")
for line in myproc.stdout:
    print(line)
print("STDERR:")
for line in myproc.stderr:
    print(line)

another possibility is that it's printing out to stderr instead of stdout.

And finally, if you call Popen with a single string argument for the command, you should also set shell=True to spawn a shell, otherwise you should use a list. Though if you do not have any arguments it can work without using the list.

Then you might be interested in the following snippet:

    p = subprocess.Popen([command], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    out = io.TextIOWrapper(p.stdout)
    err = io.TextIOWrapper(p.stderr)
    while p.poll() == None:
        ret = select.select([out.fileno(), err.fileno()], [], [])
        for fd in ret[0]:
            if fd == out.fileno():
                output_parser(out.readline())
            if fd == err.fileno():
                output_parser(err.readline())
    for line in out.read().split('\n'):
        output_parser(line)
    for line in err.read().split('\n'):
        output_parser(line)

Taken from a code I recently wrote:

https://github.com/guyzmo/pyvod/blob/master/vod/video.py#L70

HTH

this wouldn't work in my case because the script listens to real time data and never terminate unless I manually interrupt it (or kill it) — user1948847, May 20 '14 at 21:46

score 1 · Accepted Answer · answered May 20 '14 at 22:28

1

If this is linux or osx, the problem is that you need to use a pseudo-terminal to make the program think that it should operate interactively. The python pexpect module is built to handle this. It creates a pty and uses that for stdout. You can use pexpect directly or read its source to see about cooking up your own stuff. This example sends command output to the screen and to a file on disk.

import pexpect
proc = pexpect.spawn('command', logfile=open('mylog.txt','w'),
    searchwindowsize=80, timeout=-1)
for line in proc:
    print(line.strip())

answered May 20 '14 at 22:28

tdelaney

73,364
6
83
116

Unfortunately, pexpect is an external module in which i cannot use :/ – user1948847 May 21 '14 at 14:52
...but you could grab the parts you need (see openpty) in your own program. – tdelaney May 21 '14 at 15:58
@user1948847: `pexpect` is pure Python. If you can't install anything; just put it alongside your code. You could put it in a [zip archive with your code if you need a single-file executable](http://stackoverflow.com/q/5355694/4279) or use something like [PyInstaller to create a single file executable](http://www.pyinstaller.org/). – jfs May 21 '14 at 16:19

python subprocess output disappear when redirect stdout to pipe

2 Answers2