Make Perl IPC::Run pump each line, not occasional batches

Question

I use IPC::Run to get output from an external executable in a cron run script. I need it to be able to filter and make decisions based on the output on the fly. But the problem is it gives me output not on the fly, but in few batches - many lines at once, only after the executable has been run for a while. Is it possible to somehow flush the output like we can on the grep command with grep --line-buffered? I do not see this properly answered in all the Perl sites. Here is my script part:

use IPC::Run qw( start pump finish );
...
my $externalExecRun = start \@executableAndParameters, \undef, \$executableStdout, \$executableStderr ;
while (42) {
    pump $externalExecRun;
    if ($executableStdout eq '' and $engineStderr eq '') {last;}
    WriteToLog("\$executableStdout: -->$executableStdout<--");              #This writes many lines at once
    WriteToLog("\$executableStderr: -->$executableStderr<--");
    $executableStdout = "";
    $executableStderr = "";
}
finish $externalExecRun;

Please show the executable; that's the place that needs to skip buffering. — daxim, Mar 09 '18 at 13:27
The binary executable is a very unmodifable proprietary software. When it is run from bash, it gives lines one by one when they are computed, almost from start of it to the finish, continuously. But this Perl, when run from cron, and running the executable it only gets the output of that executable in big batches. Also when run with system(@executableAndParameters). — uldics, Mar 09 '18 at 13:33
https://stackoverflow.com/questions/1401002/trick-an-application-into-thinking-its-stdout-is-a-terminal-not-a-pipe https://unix.stackexchange.com/questions/249723/how-to-trick-a-command-into-thinking-its-output-is-going-to-a-terminal — daxim, Mar 09 '18 at 13:43
Re "*Is it possible to somehow flush the output like we can on the grep command with `grep --line-buffered`?*", How could we know if if you're not going to name the program? Is it even written in Perl? — ikegami, Mar 09 '18 at 17:34

score 3 · Answer 1 · answered Mar 10 '18 at 12:26

You can use IPC::Run's new_chunker to have it give you output on a line-by-line basis:

use warnings;
use strict;
use IPC::Run qw/ start new_chunker /;
use Data::Dump;

my $run = start ['perl','-le','print "a" x $_ for 1..180'],
    '>', new_chunker, \my $out, '2>', new_chunker, \my $err;
while (1) {
    $run->pump;
    last unless defined $out || defined $err;
    dd $out, $err;
    ($out,$err) = ();
}
$run->finish;

It's still possible that the external program won't output on a line-by-line basis, in which case, at least on *NIX, changing the first '>' into '>pty>' (as suggested by @ikegami in the comments) will hopefully help; or one of the links provided by @daxim.

Make Perl IPC::Run pump each line, not occasional batches

1 Answers1