1

I've been reading SO discussion on how output going to a terminal is line buffered, and output to pipes or files etc is usually "block buffered". I am trying the following two pipelines on OSX, which aim to prepend a timestamp to the incoming lines:

while [ 1 ] ; do echo $RANDOM && sleep .5 ; done | ~/bin/prepend_ts_python

and

while [ 1 ] ; do echo $RANDOM && sleep .5 ; done | ~/bin/prepend_ts_perl

The Perl program is (and I don't know much about Perl):

#!/usr/bin/perl

while ( <> ) {
    print scalar(localtime()), " ";
    print;
}

The Python program (clearly not prepending a timestamp yet) is:

#!/usr/bin/python

import sys

for line in sys.stdin:
    print("foo")

The pipeline using the Perl version prints output immediately, as desired, whereas the Python one outputs intermittently.

I was under the impression that any buffering that was going on was being determined by the while/echo loop, and for this reason people were recommending using the unbuffer command on the beginning of the pipeline.

Does Perl have some way of flushing its STDIN? I didn't think it could reach "backwards" in the pipeline. Why are they different?

Borodin
  • 126,100
  • 9
  • 70
  • 144
dcc310
  • 1,038
  • 1
  • 9
  • 13
  • The **output** is still line buffered.. – Martijn Pieters Jan 14 '17 at 19:49
  • 1
    @MartijnPieters the output to the _terminal_ is line buffered, correct? I was assuming the printing to the terminal is consistent between Python and Perl (because a standalone Python script that prints every few seconds is very real-time), and the difference in behavior I'm seeing (i.e. perl immediate, python sporadic) is due to buffering between `echo` and the scripts' input. – dcc310 Jan 14 '17 at 19:58
  • 1
    @MartijnPieters That "duplicate" doesn't really answer the question. – melpomene Jan 14 '17 at 19:59
  • 1
    Oh, it's hidden in one of the comments: "If you iterate over the input with `for line in sys.stdin: ...` then the for loop will collect a number of lines before the body of the loop is run. This will behave like buffering, though it's rather batching. Instead, do `while true: line = sys.stdin.readline()`" – melpomene Jan 14 '17 at 20:00
  • @melpomene good eyes! That indeed fixes the issue. – dcc310 Jan 14 '17 at 20:06
  • Rather than have two questions pointing to the same "answer" where the useful information is hidden in a comment, I am voting to reopen this question so that it may be answered properly. Unless someone can find a proper instance of this question being answered correctly? – Borodin Jan 14 '17 at 20:54
  • This question was closed because it was said to have been "answered" by [this comment](http://stackoverflow.com/questions/107705/disable-output-buffering#comment24604506_107717) beneath another related question. It has now been reopened, and the solution needs fleshing out and explaining by someone who know Python better than I do. – Borodin Jan 14 '17 at 21:09
  • I picked a better duplicate, one that directly points out that you should not use `sys.stdin.__iter__()`. – Martijn Pieters Jan 14 '17 at 23:05

0 Answers0