4

I have a simple script that reads values from a device and outputs them via print, and another script, which listens on stdin and interprets each number. The device outputs one number each second. Surprisingly, piping the scripts on my ubuntu box does not work. However, if the first script is made not to read from the device but generate random numbers as fast as it can, the second script successfully receives the data.

Below is a simplified example of my situation.

print.py:

#!/usr/bin/env python2
import time
import sys

while True:
    time.sleep(1)  # without this everything works
    print "42"
    sys.stdout.flush()

read.py:

#!/usr/bin/env python2
import sys

while True:
    for str in sys.stdin:
        print str

Command line invocation:

vorac@laptop:~/test$ ./print.py | ./read.py

Here is the end result. The first script reads from the device and the second graphs the data in two separate time frames (what is shown are random numbers).

enter image description here

Vorac
  • 8,726
  • 11
  • 58
  • 101
  • [this](http://stackoverflow.com/q/3670323/5128464) might be interesting for you – vlp Nov 09 '15 at 11:56
  • http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe – Karoly Horvath Nov 09 '15 at 11:57
  • I just added `newin = os.fdopen(sys.stdin.fileno(), 'r', 0); sys.stdin = newin` to `read.py` and nothing has changed. I would strongly prefer to resolve this programatically and keep the invocation of the scripts simple. – Vorac Nov 09 '15 at 12:07
  • 1
    Why don't you do both in the same script? – MKesper Nov 09 '15 at 12:23
  • @MKesper, this is a program for reading, logging and presentation of temperature data. Being able to use the modules independently seems like a good feature - you could just listen to the device, or just plot some random data, or run the interruptable logger on some other data. – Vorac Nov 09 '15 at 13:09
  • @Vorac easily solved by making each module a generator, or a separate function, or some such thing. That being said, having separate programs that each do a small task is in line with the UNIX philosophy. But in that case you could also use shell scripts or standard UNIX tools like `cat`. – David Z Nov 10 '15 at 04:12

2 Answers2

3

Ah, now that is a tricky problem. It happens because the iterator method for sys.stdin (which is xreadlines()) is buffered. In other words, when your loop implicitly calls next(sys.stdin) to get the next line of input, Python tries to read from the real under-the-hood standard input stream until its internal buffer is full, and only once the buffer is full does it proceed through the body of the loop. The buffer size is 8 kilobytes, so this takes a while.

You can see this by decreasing the time delay in the sleep() call to 0.001 or some such value, depending on the capabilities of your system. If you hit the time just right, you'll see nothing for a few seconds, and then a whole block of 42s come out all at once.

To fix it, use sys.stdin.readline(), which is unbuffered.

while True:
    line = sys.stdin.readline()
    print line

You might also want to strip off the trailing newline before printing it, otherwise you'll get double line breaks. Use line.rstrip('\n'), or just print line, to suppress the extra newline that gets printed.

David Z
  • 128,184
  • 27
  • 255
  • 279
2

I changed your read.py and it worked for me :), you forget to .readline() from stdin.

import sys
while True:
    line = sys.stdin.readline()
    if line:
        print line.strip()
    else:
        continue

Output is : 

$ python write.py | python read.py 
42
42
42
42
42
42
42
Amey Jadiye
  • 3,066
  • 3
  • 25
  • 38