When does the output of a script get appended to a file, and what can I do to circumvent the outcome?

Question

I have a script called my_bash.sh and it calls a perl script, and appends the output of the perl script to a log file. Does the file only get written once the perl script has completed?

my_bash.sh

#!/bin/bash
echo "Starting!" > "my_log.log"
perl my_perl.pl >> "my_log.log" 2>&1
echo "Ending!" >> "my_log.log"

The issue is that as the perl script is running, I'd like to manipulate contents of the my_log.log file while it's running, but it appears the file is blank. Is there a proper way to do this? Please let me know if you'd like more information.

my_perl.pl

...
foreach $component (@arrayOfComponents)
{
  print "Component Name: $component (MORE_INFO)\n";

  # Do some work to gather more info (including other prints)
  #...

  # I want to replace "MORE_INFO" above with what I've calculated here
  system("sed 's/MORE_INFO/$moreInfo/' my_log.log");
}

The sed isn't working correctly since the print statements haven't yet made it to the my_log.log.

How would you like to manipulate the output of the perl script? What have you done so far? — glenn jackman, Dec 20 '13 at 14:47
Okay, good question. I'll update my question with more info about my perl script. — ardavis, Dec 20 '13 at 14:48
you are using output redirection to redirect the output of your perl script to a logfile... that can only happen when your perl script finishes.. — Dave Lawrence, Dec 20 '13 at 14:48
When redirecting to a file, the output will happen when it happens in the perl file. Seeing that code would be helpful as well. It could be the perl file buffers all the output till the end, then prints it at one time. — Mike Gardner, Dec 20 '13 at 14:48
@daveL I'm still fairly new to scripting in general, is there a way to redirect output live instead of waiting for it to finish? — ardavis, Dec 20 '13 at 14:51
Some [discussion on flushing a file in Perl](http://stackoverflow.com/questions/4538767/how-flush-a-file-in-perl). Perl's [File::Sync](http://search.cpan.org/~cevans/File-Sync-0.09/Sync.pm) may also be useful. — Paulo Almeida, Dec 20 '13 at 14:54

score 2 · Answer 1 · answered Dec 20 '13 at 14:51

The answer to this question depends on how your my_perl.pl is outputting data and how much data is being output.

If you're using normal (buffered) I/O to produce your output, then my_log.log will only be written to once the STDOUT buffer fills. Generally speaking, if you're not producing a lot of output, this is when the program ends and the buffer is flushed.

If you're producing enough output to fill the output buffer, you will get output in my_log.log prior to my_perl.pl completing.

Additionally, in Perl, you can make your STDOUT unbuffered with the following code:

select STDOUT; $| = 1;

In which case, your output would be written to STDOUT (and then to my_log.log via redirection) the moment it is produced in your script.

I've updated my question to contain information about my perl file, thanks much! — ardavis, Dec 20 '13 at 14:51

score 2 · Accepted Answer · answered Dec 20 '13 at 14:53

Perl would buffer the output by default. To disable buffering set $| to a non-zero value. Add

$|++;

at the top of your perl script.

Quoting perldoc pervar:

$|

If set to nonzero, forces a flush right away and after every write or print on the currently selected output channel. Default is 0 (regardless of whether the channel is really buffered by the system or not; $| tells you only whether you've asked Perl explicitly to flush after each write). STDOUT will typically be line buffered if output is to the terminal and block buffered otherwise. Setting this variable is useful primarily when you are outputting to a pipe or socket, such as when you are running a Perl program under rsh and want to see the output as it's happening. This has no effect on input buffering. See getc for that. See select on how to select the output channel. See also IO::Handle.

Mnemonic: when you want your pipes to be piping hot.

score 0 · Answer 3 · answered Dec 20 '13 at 14:59

Depending on what you need to do to the log file, who might be able to read each line of output from the perl script, do something with the line, then write it to the log yourself (or not):

#!/bin/bash
echo "Starting!" > "my_log.log"
perl my_perl.pl | \
while read line; do
    # do something with the line
    echo "$line" >> "my_log.log"
done
echo "Ending!" >> "my_log.log"

score 0 · Answer 4 · answered Dec 20 '13 at 15:15

In between

print "Component Name: $component (MORE_INFO)\n";

and

system("sed 's/MORE_INFO/$moreInfo/' my_log.log");

do you print stuff? If not, delay the first print until you've figured out $moreInfo

my $header = "Component Name: $component (MORE_INFO)";
# ...
# now I have $moreInfo
$header =~ s/MORE_INFO/$moreInfo/;
print $header, "\n";

If you do print stuff, you could always "queue" it until you have the info you need:

my @output;
foreach my $component (...) {
    @output = ("Component Name: $component (MORE_INFO)");
    # ...
    push @output, "something to print"; 
    # ...
    $output[0] =~ s/MORE_INFO/$moreInfo/;
    print join("\n", @output), "\n";

I do print things, the first line is actually more like a header, and as I do some work with the component, I end up putting a statistic after it. It's meant to be a quick view kind of thing. I'm currently looking into the unbuffered output from the other answers. Hopefully it will do the trick :) — ardavis, Dec 20 '13 at 15:18

When does the output of a script get appended to a file, and what can I do to circumvent the outcome?

4 Answers4