2

I am trying to read from both stdout and stderr from a Popen and print them out. The command I am running with Popen is the following

#!/bin/bash

i=10
while (( i > 0 )); do
    sleep 1s
    echo heyo-$i
    i="$((i-1))"
done

echo 'to error' >&2

When I run this in the shell, I get one line of output and then a second break and then one line again, etc. However, I am unable to recreate this using python. I am starting two threads, one each to read from stdout and stderr, put the lines read into a Queue and another thread that takes items from this queue and prints them out. But with this, I see that all the output gets printed out at once, after the subprocess ends. I want the lines to be printed as and when they are echo'ed.

Here's my python code:

# The `randoms` is in the $PATH
proc = sp.Popen(['randoms'], stdout=sp.PIPE, stderr=sp.PIPE, bufsize=0)

q = Queue()

def stream_watcher(stream, name=None):
    """Take lines from the stream and put them in the q"""
    for line in stream:
        q.put((name, line))
    if not stream.closed:
        stream.close()

Thread(target=stream_watcher, args=(proc.stdout, 'out')).start()
Thread(target=stream_watcher, args=(proc.stderr, 'err')).start()

def displayer():
    """Take lines from the q and add them to the display"""
    while True:
        try:
            name, line = q.get(True, 1)
        except Empty:
            if proc.poll() is not None:
                break
        else:
            # Print line with the trailing newline character
            print(name.upper(), '->', line[:-1])
            q.task_done()

    print('-*- FINISHED -*-')

Thread(target=displayer).start()

Any ideas? What am I missing here?

sharat87
  • 7,330
  • 12
  • 55
  • 80
  • What you are missing is that the pipeline is buffered. I don't know how you can make the bash process flush the buffer, though. – Sven Marnach Apr 01 '12 at 12:04
  • Also, I don't think that stream watcher works. `for line in stream` will exhaust whatever lines are initially available then close the stdout / stderr -- I don't think you want to close them, first of all, and you're not waiting and re-checking for more lines to appear. – agf Apr 01 '12 at 12:07
  • @SvenMarnach, pipeline is buffered by python? If so, how can I tell it to not do that and give me the data instead. @agf, The for loop won't end until a `StopIteration` is raised, which I believe happens when the subprocess closes its side of the pipe. Which is why I close the stream immediately. Besides, I'm checking if the process is running in the displayer thread. – sharat87 Apr 01 '12 at 12:12
  • The buffering is performed by the operating system, and you will have a hard time to prevent it. See [Force another program's standard output to be unbuffered using Python](http://stackoverflow.com/questions/1544050/force-another-programs-standard-output-to-be-unbuffered-using-python) for related information. – Sven Marnach Apr 01 '12 at 13:17
  • If its buffered by the operating system, how is my bash shell able to print output immediately as it is available? That's the reason I believe this is possible with python's subprocess too. – sharat87 Apr 01 '12 at 13:22
  • If stdout is a terminal, it will be line-buffered, so as soon as a full line was printed, it will show up. – Sven Marnach Apr 01 '12 at 19:31
  • As tchrist stated in his excellent answer, I was wrong about the buffering being done by the OS -- it's done by the C library. – Sven Marnach Apr 01 '12 at 22:57

2 Answers2

4

Only stderr is unbuffered, not stdout. What you want cannot be done using the shell built-ins alone. The buffering behavior is defined in the stdio(3) C library, which applies line buffering only when the output is to a terminal. When the output is to a pipe, it is pipe-buffered, not line-buffered, and so the data is not transferred to the kernel and thence to the other end of the pipe until the pipe buffer fills.

Moreover, the shell has no access to libc’s buffer-controlling functions, such as setbuf(3) and friends. The only possible solution within the shell is to launch your co-process on a pseudo-tty, and pty management is a complex topic. It is much easier to rewrite the equivalent shell script in a language that does grant access to low-level buffering features for output streams than to arrange to run something over a pty.

However, if you call /bin/echo instead of the shell built-in echo, you may find it more to your liking. This works because now the whole line is flushed when the newly launched /bin/echo process terminates each time. This is hardly an efficient use of system resources, but may be an efficient use of your own.

tchrist
  • 78,834
  • 30
  • 123
  • 180
  • Excellent answer. Thank you very much. Yes, I don't want to go though managing a pty. I didn't know stdio's buffering is dependent on the target being written to. That's a very useful piece of information. – sharat87 Apr 02 '12 at 04:45
  • 1
    `expect` includes an `unbuffer(1)` utility, that does the pty dance to disable stdio's buffering. So the OP could just wrap the `randoms` script in another script that just does (warning: untested) `randoms | unbuffer -p`. – ninjalj Apr 02 '12 at 18:33
  • I've tried a lot of approaches like making a non-blocking `stdout` by `fcntl.fcntl(..., os.O_NONBLOCK)`, but the only solution which helped me was described [here](https://stackoverflow.com/a/5413588/676369). In short, the solution is to make a non-blocking `stdout` using `pty.openpty()` – Exterminator13 Oct 22 '17 at 12:17
-1

IIRC, setting shell=True on Popen should do it.

aquavitae
  • 17,414
  • 11
  • 63
  • 106
  • Nope, didn't do it. Of course, with this argument, I changed `['randoms']` to `'randoms'`. But, yeah, I don't see any output until the subprocess finishes. – sharat87 Apr 01 '12 at 13:45
  • @ShrikantSharat: if you have `expect` installed, you should be able to set `shell=True` and launch `randoms | unbuffer -p`. – ninjalj Apr 02 '12 at 18:34
  • @ninjalj, That's a useful tool. Thanks. Could you put that as an answer, for the record? – sharat87 Apr 03 '12 at 05:06