stdout to file1, stderr to file2, both correctly interleaved to stdout and file

Question

Given a third-party program, how would one simultaneously:

write stdout to z.stdout
write stderr to z.stderr
pass exit codes appropriately
write both in correct interleaved order to stdout

Here's the test program (delayed_interleaved_stdout_stderr.pl) I've been using:

#!/usr/bin/env perl

use strict;
use warnings;

# fixme: debug, uncomment to force stdout flushing
# use English '-no_match_vars';
# $OUTPUT_AUTOFLUSH = 1;

# use sleeps to simulate delays and test buffering
use Time::HiRes 'sleep';

foreach my $num ( 0..9 ) {
  if ( 0 == $num % 2 ) {
    print STDOUT $num, ":stdout\n";
  }
  else {
    print STDERR $num, ":stderr\n";
  }
  sleep 0.25;
}

So far I've been able to do 1,2,3 with:

( set -o pipefail; \
  ( set -o pipefail; delayed_interleaved_stdout_stderr.pl \
    | tee z.stdout; exit $? \
  ) 3>&1 1>&2 2>&3 | tee z.stderr; exit $? \
) 3>&1 1>&2 2>&3

Thanks to a related answer by lhunath and a friend, I simplified it to:

delayed_interleaved_stdout_stderr.pl > >(tee z.stdout) 2> >(tee z.stderr >&2)

However, I haven't been able to get correct interleaved order. stderr prints immediately and the (presumably buffered) stdout all prints at the end.

1:stderr
3:stderr
5:stderr
7:stderr
9:stderr
0:stdout
2:stdout
4:stdout
6:stdout
8:stdout

Running delayed_interleaved_stdout_stderr.pl by itself displays in the proper 0-9 order. Forcing stdout to flush works appropriately (see commented fixme section), but I won't be able to modify the real files.

Maybe I'm missing something basic and I'm beginning to wonder if this is possible at all :(

The bash snippets are fine - it seems to me the issue is with perl's output buffering. If you can figure out a way to unbuffer the output without modifying the script then the order should be correct. I tried using the GNU `stdbuf` utility but it didn't do anything. — jw013, Aug 12 '11 at 02:52
doesn't seem likely that it will help, but see if your system has the `unbuffer` cmd. It's worth a try. Is the unmodifiable code perl? Good luck. — shellter, Aug 12 '11 at 16:41
At least some of it is Perl; I'm going to work with the author to force stdout flushing wherever possible. Thanks for the suggestion! — vlee, Aug 13 '11 at 20:25
By the way, if I'm going to go with: "delayed_interleaved_stdout_stderr.pl > >(tee z.stdout) 2> >(tee z.stderr >&2)" and try to force stdout flushing, should I update my question and answer my own question, or is it better to leave this unanswered? — vlee, Aug 13 '11 at 20:25

score 2 · Answer 1 · answered Aug 12 '11 at 17:01

2

Requirement 4 is the tricky one. The buffering logic for stdout and stderr is buried within libc, and to change it you need to fool the application to think it's writing to a terminal.

unbuffer which comes with the expect package will do this for you. Caveat: even writing to a terminal, stdout is line-buffered, so if your app isn't writing full lines then this won't work.

answered Aug 12 '11 at 17:01

PhilR

5,375
1
21
27

I removed requirement #4 (interleaved to file) and moved #5 to #4. I also confirmed I can enforce stdout flushing for the third-party programs! /cheer – vlee Aug 15 '11 at 17:57

score 1 · Accepted Answer · answered Aug 15 '11 at 17:59

1

I confirmed I can enforce stdout flushing for the third-party programs. In light of this, I'm going with

delayed_interleaved_stdout_stderr.pl > >(tee z.stdout) 2> >(tee z.stderr >&2)

Thanks for all help!

answered Aug 15 '11 at 17:59

vlee

1,369
3
14
23

stdout to file1, stderr to file2, both correctly interleaved to stdout and file

2 Answers2