1

My program (a "TRAC Processor") uses character-by-character input. I am implementing readline-like input features for strings which are terminated with characters other than enter (usually ') and may themselves be multi-line. So I output terminal escape sequences between input characters, including escape sequences which query the terminal emulator (cursor position and screen size). To do cross-platform single-character input, I used http://code.activestate.com/recipes/134892/, which was very helpful.

This worked fine with paste... until I needed to get a terminal response after the first character of the paste. It seemed like the pasted text was getting mingled with the response to the escape sequence. I thought I would fix it by flushing the input buffer before initiating the escape-sequence query: wait 10ms, and if there is no input proceed; if there is input, buffer it and wait again until no input. Based on this post, I tried to poll stdin using select(). Great idea, but it didn't work, and it produced very strange behavior. I posted that strange behavior in the original version of this question, thinking I was misunderstanding select and there was a way to fix it. There doesn't seem to be, but I have found another way to flush (and save) the input stream. I decided to keep this question, and post that method as answer.

The problem with select() is explained here. After the first character of the paste, the other characters are already buffered, and select only returns new input when there is new input beyond what is already buffered. I couldn't bring myself to delete the MWE I'd produced of this behavior, so you can see it below.

Unfortunately, the answers proposed in that post either don't work for me or need a lot more explanation. @slowdog suggests using unbuffered input (os.read(stdin.fileno(), 1) instead of stdin.read(1)). That solves the select problem, but it breaks paste: it seems that all the characters of the paste after the first one are buffered no matter what, so you never see them. It also didn't seem to work well with the escape-sequence responses, which seem to also get buffered. It's also annoying because you need to flush the output buffer, but that's not so terrible. @Omnafarious, in a comment, said "Though, another way to handle the Python buffering issue to to simply do a no-parameter read, which should read everything currently available." That is ultimately what I did, as posted below, but "simply" turns out not to be so simple. There is another solution here, but I figured there must be a way to do this without threading.

Incidentally, there is a relatively simple work-around, because it turns out that the paste is not randomly interspersed with response to the escape sequence. The entire remainder of the paste gets read before the escape sequence response, so that when you are looking for the escape-sequence response (which itself starts with an escape), you can just buffer all the characters you read before the escape, and process them later. This only fails if you might be typing ESC characters in at the terminal. In any case, by this time I was pretty much hooked on solving this problem, and I thought others might find the answer valuable.

Anyway, FWIW here is my MWE for the select problem, which just echoes the text rather than buffering it:

def flush():
    import sys, tty, termios
    from select import select
    tty.setraw(sys.stdin.fileno())
    while True:
        rlist, wlist, xlist = select([sys.stdin], [], [], 1)
        if rlist == []: return
        sys.stdout.write(sys.stdin.read(1))

Paste this into the Python prompt (2.7.9) and put another blank line at the end. If you invoke flush() and type some text more quickly than one letter per second, it types it back to you. For example, I typed "hello" and then paused, and got this result:

>>> flush()
hello>>>

In the OSX Terminal app (at least), if you copy the word text to the clipboard, invoke the function and hit paste within one second, here's what you get:

>>> flush()
t>>>

Odd! Only the first letter. Try it again, typing nothing:

>>> flush()
>>>

It paused for a second and does nothing, like no input waiting, right? Try it again, and hit ?:

>>> flush()
ext?>>>

You get the rest of the paste, saved up, before the ?!! Also, strangely, there is a 1-second pause before it types the ? which I don't understand. If you try again at this point, it behaves like normal.

OK, let's try it again, first pasting text, then pasting WTF, then typing !:

>>> flush()
t>>> flush()
extW>>> flush()
TF!>>>

So again the paste only gives the first letter, and holds the others in the input buffer, and pauses for a second before W and !. Yet another strange thing: the buffered characters are not entered at the Python >>> prompt.

One lingering question: why do you get the additional 1-second pause before the next letter is echoed? Select does not always wait for the whole time period...

Community
  • 1
  • 1
Nat Kuhn
  • 9,493
  • 5
  • 18
  • 26

1 Answers1

0

The "no-parameter read" is often cited as a way to read all available bytes, which sounds perfect for this application. Unfortunately, when you look at the documentation, read() is the same as readall(), which blocks until EOF. So you need to set stdin to non-blocking mode.

Once you do this, you start getting:

IOError: [Errno 35] Resource temporarily unavailable

When you google this, the vast majority of responses say that the solution to this problem is to get rid of non-blocking mode... so that's not helpful here. However, this post explains that this is simply what the non-blocking read() does when there are no characters to return.

Here is my flush() function, much of which copied from that post:

def flush():
    import sys, tty, termios, fcntl, os
    fd = sys.stdin.fileno()
    old_attr = termios.tcgetattr(fd)
    old_fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    try:
        tty.setraw(fd)
        fcntl.fcntl(fd, fcntl.F_SETFL, old_fl | os.O_NONBLOCK)
        inp = sys.stdin.read()
    except IOError, ex1:  #if no chars available generates exception
        try: #need to catch correct exception
            errno = ex1.args[0] #if args not sequence get TypeError
            if errno == 35:
                return '' #No characters available
            else:
                raise #re-raise exception ex1
        except TypeError, ex2:  #catch args[0] mismatch above
            raise ex1 #ignore TypeError, re-raise exception ex1
    finally:
        termios.tcsetattr(fd, termios.TCSADRAIN, old_attr)
        fcntl.fcntl(fd, fcntl.F_SETFL, old_fl)
    return inp

Hope it's helpful to someone!

Nat Kuhn
  • 9,493
  • 5
  • 18
  • 26