Shell pipe working asynchronously

Question

I thought each processes pipe connected work asynchronously but it doesn't.

a.py

#!/usr/bin/env python
import sys
import time
for line in sys.stdin:
    time.sleep(2)
    sys.stdout.write(line.upper())
    sys.stdout.flush()

and b.py

#!/usr/bin/env python
import sys
for line in sys.stdin:
    sys.stdout.write(line.capitalize())
    sys.stdout.flush()

and test.txt

hello
world
python

Following code show each lines one by one by 2 seconds.

$ ./a.py < test.txt
HELLO
WORLD
PYTHON

But following code show entirely one time.

$ ./a.py < test.txt | ./b.py
Hello
World
Python

It looks that shell pipe is working synchronously. How can I do asynchronously?

score 0 · Answer 1 · answered May 15 '13 at 09:42

0

You could use threads, but I don't know if this fits your likings. ;)

Ps: thread in python are very easy, I can provide an example, if you need.

answered May 15 '13 at 09:42

user2384980

1
1

score 0 · Accepted Answer · answered May 15 '13 at 11:23

The problem with the code is that, although the expression...

for line in sys.stdin:
    # ...

...is treating sys.stdin as an iterator, it doesn't seem to yield any values until sys.stdin hits EOF, much like the behavior of sys.stdin.readlines().

For the first case, it hits EOF almost immediately, but in the second case, it has to wait for a.py to terminate, which takes 6 seconds.

You'll have to rewrite the code for b.py like this...

#!/usr/bin/env python
import sys
while 1:
    line = sys.stdin.readline()
    if not line:
        break
    sys.stdout.write(line.capitalize())
    sys.stdout.flush()

...using readline() to get the call to return as soon as it has a whole line, rather than waiting for EOF.

You should probably also make a similar change to a.py in case you want to pipe input from something other than a text file.

Thanks for your answer. I found similar thread in http://stackoverflow.com/a/7608205/395425 In python2.7 "for" statement is waiting the EOF, but python3.3 doesn't wait. — Hyungyong Kim, May 15 '13 at 14:50
I found later, python 2.x "for line in sys.stdin" does not wait to EOF. It seems to be buffering. — Hyungyong Kim, May 20 '13 at 09:12

Shell pipe working asynchronously

2 Answers2