how to read an incomplete line from a Python pty.fork() child output?

Question

#!/usr/bin/python

import pty
import os
import sys

pid, out = pty.fork()

if pid:
    try:
        for line in os.fdopen(out):
            sys.stdout.write(line)
    except IOError:
        pass

else:
    sys.stdout.write("foobar")
    sys.stdout.flush()

prints nothing. How do I make it print the incomplete line emitted by the child??

Running this prints `foobar` for me. Maybe add a `sys.stdout.flush()` in the else block? — Ismail Badawi, Sep 15 '14 at 22:50
@IsmailBadawi definitely prints nothing for me, just double checked. I run Python 2.7. Let me try with flush — Mark Galeck, Sep 15 '14 at 22:53
Shouldn't you actually close the stream at the writer's end? How else could Python's input routine (probably implemented via `getline`) possibly know whether it is encountering an incomplete line or whether more characters are to be written? — 5gon12eder, Sep 15 '14 at 22:57
@MarkGaleck I am having no problem seeing "foobar" on my end — asdf, Sep 15 '14 at 22:58
@5gon12eder can't - the problem is, I am using it in the situation, when the child may die, while emitting a line — Mark Galeck, Sep 15 '14 at 22:58
Are you running this function from the command line? If so are you on linux, windows or a mac? — asdf, Sep 15 '14 at 22:59
@BadKarma I see two people without any problems, and I am definitely not seeing it. This must be either a Python version, or some environmental setting? ID10T I think is out of the question this time, since I pasted the code for you. What environmental condition should I be checking for? — Mark Galeck, Sep 15 '14 at 23:00
@BadKarma I am on Linux, a version of Fedora, yes on command line, I just do >./foobar.py — Mark Galeck, Sep 15 '14 at 23:00
@5gon12eder you have a point. How could it know? I don't know. But, for BadKarma and Ismail, it does somehow "know". Maybe we ask them. — Mark Galeck, Sep 15 '14 at 23:05
I don't see any output either (GNU/Linux). I changed `except IOError: pass` to `except IOError as e: print(e)` and it tells me `[Errno 5] Input/output error`. — 5gon12eder, Sep 15 '14 at 23:06
@5gon12eder oh, well, I was expecting originally, that in the exception code, I would be able to recover the incomplete line - the exception code is called, upon the child dying, so then Python knows, there is no more output and could possibly collect the incomplete line. Alas, I don't know if that is possible, or if so, how to do it from the exception handler. — Mark Galeck, Sep 15 '14 at 23:06
Yes so me and 5gon12eder see it one way, exactly the same. And BadKarma and Ismail, see it the other way. Why would that be? — Mark Galeck, Sep 15 '14 at 23:07
@MarkGaleck The reason I could see it is because I was running it on my mac. I read up on `sys` tools and they're highly platform dependent and not necessarily supported across all of them. I've been doing some research on a work-around and am making some progress. I'll post an answer if I find something that works — asdf, Sep 15 '14 at 23:10
@BadKarma gee, thank you so much, can I buy you lunch or something :)? — Mark Galeck, Sep 15 '14 at 23:11
@MarkGaleck Found this resource. Maybe give it a peek? http://stackoverflow.com/questions/10409897/ioerror-input-output-error-when-printing — asdf, Sep 15 '14 at 23:25

5gon12eder · Accepted Answer · 2014-09-16T00:11:10.940

The following is pure speculation: Iterating over an input stream is likely implemented as reading characters into a buffer until either a newline or an end-of-file condition is encountered. If the child dies, some implementations (platform dependent) loose the remaining characters from the buffer.

Maybe using some more low-level I/O can avoid this issue. When I run the original script on my GNU/Linux system, I don't get the “foobar” but an IOError instead. However, when I change it to

with os.fdopen(out) as istr:
    sys.stdout.write(istr.read())

it prints “foobar” without throwing any exception.

Update: In order to read the stream one piece at a time, we'll need to resort to even more low-level I/O. I found that the following works for me:

import pty
import os
import sys

pid, out = pty.fork()

if pid:
    while True:
        buffsz = 10  # Use a larger number in production code.
        buff = b''
        died = False
        try:
            buff = os.read(out, buffsz)
        except IOError:
            died = True
        sys.stdout.write(buff.decode())
        if len(buff) == 0 or died:
            break
else:
    with sys.stdout:
        # Also try writing a longer string with newlines.
        sys.stdout.write("foobar")

Unfortunately, this means we'll need to reassemble the buffer chunks manually and scan for newlines. This is inconvenient but certainly can be done.

Yes, but... of course, it reads the whole available output at once. If the output is ongoing and voluminous, I think this would not work. I tried to use readline() here, does not work as before. — Mark Galeck, Sep 15 '14 at 23:39
I am sorry I don't want to, but I have to unaccept. Because I wanted to read the incomplete line. As it is, it reads the whole thing at once, not line-by-line. — Mark Galeck, Sep 15 '14 at 23:43
@MarkGaleck I have improved a bit on the solution. Hope it is more useful to you now. — 5gon12eder, Sep 16 '14 at 00:04

how to read an incomplete line from a Python pty.fork() child output?

1 Answers1