1

This very well may fall under KISS (keep it simple) principle but I am still curious and wish to be educated as to why I didn't receive the expected results. So, here we go...

I have a shell script to capture STDOUT and STDERR without disturbing the original file descriptors. This is in hopes of preserving the original order of output (see test.pl below) as seen by a user on the terminal.

Unfortunately, I am limited to using sh, instead of bash (but I welcome examples), as I am calling this from another suite and I may wish to use it in a cron in the future (I know cron has the SHELL environment variable).

wrapper.sh contains:

#!/bin/sh
stdout_and_stderr=$1
shift
command=$@

out="${TMPDIR:-/tmp}/out.$$"
err="${TMPDIR:-/tmp}/err.$$"
mkfifo ${out} ${err}
trap 'rm ${out} ${err}' EXIT
> ${stdout_and_stderr}
tee -a ${stdout_and_stderr} < ${out} &
tee -a ${stdout_and_stderr} < ${err} >&2 &
${command} >${out} 2>${err}

test.pl contains:

#!/usr/bin/perl

print "1: stdout1\n";
print STDERR "2: stderr1\n";
print "3: stdout2\n";

In the scenario:

sh wrapper.sh /tmp/xxx perl test.pl

STDOUT contains:

1: stdout1
3: stdout2

STDERR contains:

2: stderr1

All good so far...

/tmp/xxx contains:

2: stderr1
1: stdout1
3: stdout2

However, I was expecting /tmp/xxx to contain:

1: stdout1
2: stderr1
3: stdout2

Can anyone explain to me why STDOUT and STDERR are not appending /tmp/xxx in the order that I expected? My guess would be that the backgrounded tee processes are blocking the /tmp/xxx resource from one another since they have the same "destination". How would you solve this?

related: How do I write stderr to a file while using "tee" with a pipe?

Andrew Sohn
  • 675
  • 1
  • 6
  • 15
  • Whatever you do here, if one of the `tee`s gets blocked for a while (because of heavy load on the system, or whatever), the data that goes through it will also be delayed. There's really no way around that, as the output from the process contains no ordering or timestamps that could be used to arrange the writes correctly at the other end. – ilkkachu May 21 '18 at 22:24

2 Answers2

3

It is a feature of the C runtime library (and probably is imitated by other runtime libraries) that stderr is not buffered. As soon as it is written to, stderr pushes all of its characters to the destination device.

By default stdout has a 512-byte buffer.

The buffering for both stderr and stdout can be changed with the setbuf or setvbuf calls.

From the Linux man page for stdout:

NOTES: The stream stderr is unbuffered. The stream stdout is line-buffered when it points to a terminal. Partial lines will not appear until fflush(3) or exit(3) is called, or a newline is printed. This can produce unexpected results, especially with debugging output. The buffering mode of the standard streams (or any other stream) can be changed using the setbuf(3) or setvbuf(3) call. Note that in case stdin is associated with a terminal, there may also be input buffering in the terminal driver, entirely unrelated to stdio buffering. (Indeed, normally terminal input is line buffered in the kernel.) This kernel input handling can be modified using calls like tcsetattr(3); see also stty(1), and termios(3).

wallyk
  • 56,922
  • 16
  • 83
  • 148
1

After a little bit more searching, inspired by @wallyk, I made the following modification to wrapper.sh:

#!/bin/sh
stdout_and_stderr=$1
shift
command=$@

out="${TMPDIR:-/tmp}/out.$$"
err="${TMPDIR:-/tmp}/err.$$"
mkfifo ${out} ${err}
trap 'rm ${out} ${err}' EXIT
> ${stdout_and_stderr}
tee -a ${stdout_and_stderr} < ${out} &
tee -a ${stdout_and_stderr} < ${err} >&2 &
script -q -F 2 ${command} >${out} 2>${err}

Which now produces the expected:

1: stdout1
2: stderr1
3: stdout2

The solution was to prefix the $command with script -q -F 2 which makes script quite (-q) and then forces file descriptor 2 (STDOUT) to flush immediately (-F 2).

I am now researching to determine how portable this is. I think -F pipe may be Mac and FreeBSD and -f or --flush may be other distros...

related: How to make output of any shell command unbuffered?

Andrew Sohn
  • 675
  • 1
  • 6
  • 15
  • now i'm struggling with argument expansion from the calling script. i'm attempting to write the command to a tmp file and have wrapper.sh read it but this in just becoming more trouble than it's worth. maybe someone will find another use for this though – Andrew Sohn May 21 '18 at 21:03