How do I write a unix filter in python?

Question

I want to write a program that reads stdin (unbuffered) and writes stdout (unbuffered) doing some trivial char-by-char transformation. For the sake of the example let's say I want to remove all chars x from stdin.

Answer 1

You can use the fileinput class, which lets you process inputs like the Perl diamond operator would. From the docs:

import fileinput
for line in fileinput.input():
    process(line)

where process does something like print line.replace('x','').

You can follow this StackOverflow question for how to unbuffer stdout. Or you can just call sys.stdout.flush() after each print.

Answer 2

Read from sys.stdin and write to sys.stdout (or use print). Your example program:

import sys

for line in sys.stdin:
    print line.replace("x", ""),

There isn't a standard way to make stdin unbuffered, and you don't want that. Let the OS buffer it.

Answer 3

I don't know exactly what you mean by buffered in this context, but it is pretty simple to do what you are asking...

so_gen.py (generating a constant stream that we can watch):

import time
import sys
while True:
    for char in 'abcdefx':
        sys.stdout.write(char)
        sys.stdout.flush()
        time.sleep(0.1)

so_filter.py (doing what you ask):

import sys
while True:
    char = sys.stdin.read(1)
    if not char:
        break
    if char != 'x':
        sys.stdout.write(char)
        sys.stdout.flush()

Try running python so_gen.py | python so_filter.py to see what it does.

Answer 4

Use the -u switch for the python interpreter to make all reads and writes unbuffered. Similar to setting $| = true; in Perl. Then proceed as you would, reading a line modifying it and then printing it. sys.stdout.flush() not required.

#!/path/to/python -u

import sys

for line in sys.stdin:
    process_line(line)