Send standard err to pipe

Question

I am trying to write a simple script that takes standard output and standard err and puts the word STDERR: at the beginning of each line of standard err. For testing I have a simple script that outputs a couple lines alternating between standard out and standard err:

#!/usr/bin/perl
print "OUT 1\n";
print STDERR "ERR 1\n";
print "OUT 2\n";
print STDERR "ERR 2\n";

When I run it:

lorkenpeist$ ./testscript.pl
OUT 1
ERR 1
OUT 2
ERR 2

And here is my script stderr.awk to add STDERR:

#!/bin/awk -f
{print "STDERR: " $0}

If I run ./testscript.pl | ./stderr.awk (which is obviously wrong because I'm piping standard out instead of standard err):

lorkenpeist$ ./testscript.pl | ./stderr.awk
ERR 1
ERR 2
STDERR: OUT 1
STDERR: OUT 2

I see that standard err is output immediately, while standard output is delayed because of the pipe. The original order of the print statements is not preserved.

I can also redirect standard err to standard output:

lorkenpeist$ ./testscript.pl 2>&1 | ./stderr.awk
STDERR: ERR 1
STDERR: ERR 2
STDERR: OUT 1
STDERR: OUT 2

Not only is everything processed by stderr.awk instead of just standard err, but again the order of the print statements is not preserved. Is there any way to send just the standard err to stderr.awk, and also preserve the order of the print statements? What I'd really like to see is:

OUT 1
STDERR: ERR 1
OUT 2
STDERR: ERR 2

I'm beginning to suspect that IO redirection simply isn't the answer, but I'm at a loss for alternatives.

EDIT: Given that standard output is buffered and standard err is not, It looks like I don't have complete control over the order in which the print statements appear on the terminal. That being said, I would prefer if order was at least somewhat preserved, instead of all of standard err being printed before any of standard output. Alternatively, is there a way to make standard output and/or pipes unbuffered?

Answer 1

Like Andy Lutomirski, I think that How to pipe stderr and not stdout can help.

Here's a command sequence that filters standard error without changing standard output; ./genouterr.sh generates information on both standard output and standard error.

( ./genouterr.sh 2>&1 1>&3 | sed 's/^/STDERR: /' >&2) 3>&1

What that does is:

• Run a subshell (...) and sends it output from file descriptor 3 to standard output (3>&1).
• In the subshell, it runs the command ./genouterr.sh, piping its standard output to sed.
• But, before running the shell script, it arranges for error output to go to standard output (2>&1), and for standard output to go to file descriptor 3 (1>&3). Thus, anything written to standard output by the script goes to file descriptor 3 in fact, while anything written to standard error by the script goes down the pipe.
• The RHS of the pipeline is sed, inserting STDERR: at the start of each line, and then redirecting its standard output to standard error (1>&2).

The net result is that what ./genouterr.sh writes on standard output appears on standard output; what ./genouterr.sh writes on standard error is prefixed with STDERR: and appears on standard error.

Note that because of buffering and so on, the output from the commands may interleave standard output and standard error differently from the way they'd appear on the terminal screen without any I/O redirection. This is pretty close to unavoidable. If you are going to avoid it, you get involved in using pseudo-ttys (ptys), which is a complex area, and I'd not want to guarantee the behaviour even then if you need to filter something through sed.

The script genouterr.sh simply generates both standard output and standard error lines:

#!/bin/bash
for i in {01..50}
do
  echo "stdout $i"
  echo "stderr $i" >&2
done

And the original test I did (I wrote this a month or two ago, as an exercise extending the other question) was looking for standard error lines containing a number ending in 0 (using grep); changing that to a sed script is a work of moments.

#!/bin/bash
set -x
rm -f out.[123]
./genouterr.sh 1>/dev/null
./genouterr.sh 2>/dev/null
( ./genouterr.sh 2>&1 1>&3 | grep '[0-9]0' >&2) 3>out.3 2>out.2 1>out.1
ls -l out.[123]
( ./genouterr.sh 2>&1 1>&3 | grep '[0-9]0' >&2) 3>&1

When you run this script, you find that the file out.1 is empty, out.3 contains what genouterr.sh wrote to standard output, and out.2 contains the filtered version of what genouterr.sh wrote to standard error.

Answer 2

The answers from How to pipe stderr, and not stdout? may be useful. Your best bet is probably something like:

(echo 'out'; echo 'err' >&2; echo 'out2'; echo 'err2' >&2) 2> >(sed -e 's/^/STDERR: /' >&2)

This has synchronicity issues, though -- the last couple STDERR lines will be written after bash thinks the command is done.

To improve on it, you could kick off the sed command as a separate asynchronous process and then explicitly wait for it, but this will quickly get messy.

You'll need to be careful, though: libc won't buffer writes to stderr. For more deterministic behavior, you can make sure that stdout is a tty by creating a pseuto-tty (see the manpage for pty for details -- there's little in the way of documentation for this). For that fancy stuff, though, you'll probably have to use a real programming language (e.g. C or Python).