13

I've wrote this simple script, it generates one output line per second (generator.sh):

for i in {0..5}; do echo $i; sleep 1; done

The raku program will launch this script and will print the lines as soon as they appear:

my $proc = Proc::Async.new("sh", "generator.sh");
$proc.stdout.tap({ .print });
my $promise = $proc.start;
await $promise;

All works as expected: every second we see a new line. But let's rewrite generator in raku (generator.raku):

for 0..5 { .say; sleep 1 }

and change the first line of the program to this:

my $proc = Proc::Async.new("raku", "generator.raku");

Now something wrong: first we see first line of output ("0"), then a long pause, and finally we see all the remaining lines of the output.

I tried to grab output of the generators via script command:

script -c 'sh generator.sh' script-sh
script -c 'raku generator.raku' script-raku

And to analyze them in a hexadecimal editor, and it looks like they are the same: after each digit, bytes 0d and 0a follow.

Why is such a difference in working with seemingly identical generators? I need to understand this because I am going to launch an external program and process its output online.

fingolfin
  • 591
  • 9

2 Answers2

19

Why is such a difference in working with seemingly identical generators?

First, with regard to the title, the issue is not about the reading side, but rather the writing side.

Raku's I/O implementation looks at whether STDOUT is attached to a TTY. If it is a TTY, any output is immediately written to the output handle. However, if it's not a TTY, then it will apply buffering, which results in a significant performance improvement but at the cost of the output being chunked by the buffer size.

If you change generator.raku to disable output buffering:

$*OUT.out-buffer = False; for 0..5 { .say; sleep 1 }

Then the output will be seen immediately.

I need to understand this because I am going to launch an external program and process its output online.

It'll only be an issue if the external program you launch also has such a buffering policy.

Jonathan Worthington
  • 29,104
  • 2
  • 97
  • 136
1

In addition to answer of @Jonathan Worthington. Although buffering is an issue of writing side, it is possible to cope with this on the reading side. stdbuf, unbuffer, script can be used on linux (see this discussion). On windows only winpty helps me, which I found here.

So, if there are winpty.exe, winpty-agent.exe, winpty.dll, msys-2.0.dll files in working directory, this code can be used to run program without buffering:

my $proc = Proc::Async.new(<winpty.exe -Xallow-non-tty -Xplain raku generator.raku>);
fingolfin
  • 591
  • 9