Pipe only STDERR through a filter

Question

Is there any way, in bash, to pipe STDERR through a filter before unifying it with STDOUT? That is, I want

STDOUT ────────────────┐
                       ├─────> terminal/file/whatever
STDERR ── [ filter ] ──┘

rather than

STDOUT ────┐
           ├────[ filter ]───> terminal/file/whatever
STDERR ────┘

See also [How to pipe stderr, and not stdout?](http://stackoverflow.com/questions/2342826/how-to-pipe-stderr-and-not-stdout). — Andrew Marshall, May 22 '13 at 01:44
This deserves an upvote simply because it includes a beautiful ASCII-art diagram. — Leif Wickland, Mar 12 '21 at 15:52

score 71 · Accepted Answer · edited Nov 27 '15 at 06:45

71

Here's an example, modeled after how to swap file descriptors in bash . The output of a.out is the following, without the 'STDXXX: ' prefix.

STDERR: stderr output
STDOUT: more regular

./a.out 3>&1 1>&2 2>&3 3>&- | sed 's/e/E/g'
more regular
stdErr output

Quoting from the above link:

First save stdout as &3 (&1 is duped into 3)

Next send stdout to stderr (&2 is duped into 1)

Send stderr to &3 (stdout) (&3 is duped into 2)

close &3 (&- is duped into 3)

edited Nov 27 '15 at 06:45

evandrix

6,041
4
27
38

answered Sep 01 '10 at 13:08

Paul Rubel

26,632
7
60
80

2

It doesn't work for me. Finally I make it work with `3>&2 2>&1 1>&3-`. – aleung Jan 25 '18 at 10:37
6

**Caution**: this assumes FD 3 is not in use, and doesn't undo the swapping of file descriptors 1 and 2, so you can't go on to pipe this to yet another command. See [this answer](https://stackoverflow.com/a/52575213/5353461) for further detail and work-around. For a much cleaner syntax for {ba,z}sh, see [this answer](https://stackoverflow.com/a/52575087/5353461). – Tom Hale Sep 30 '18 at 06:24

score 68 · Answer 2 · edited Jun 20 '20 at 09:12

68

TL;DR:

$ cmd 2> >(stderr-filter >&2)

Example:

% cat /non-existant 2> >(tr o X >&2)
cat: /nXn-existant: NX such file Xr directXry
%

This will work in both bash and zsh. Bash is pretty much ubiquitous these days, however, if you really do need a (really gnarly) solution for POSIX sh, then see here.

Explanation

By far, the easiest way to do this is to redirect STDERR via process substitution:

Process substitution allows a process’s input or output to be referred to using a filename. It takes the form of
>(list)
The process list is run asynchronously, and its input or output appears as a filename.

So what you get with process substituion is a filename.

Just like you could do:

$ cmd 2> filename

you can do

$ cmd 2> >(filter >&2)

The >&2 redirect's filter's STDOUT back to the original STDERR.

edited Jun 20 '20 at 09:12

Community

1
1

answered Sep 30 '18 at 06:00

Tom Hale

40,825
36
187
242

6

There is a caveat that goes along this answer: bash _does not wait for the substituted process to complete_, while the FD juggling to swap 1 and 2 does. This may be important. I got burned by this doing `exec 2> >(while .. read .. echo)` in a script running as a systemd service. journald captures fd2 of the service, and infers log level from an '' prefix: prepend `<2> ` to output lines, and you get ERROR level assigned to journal records. But sometimes systemd was swifter to kill the whole process group upon exit of the main one before it could log its last, the most important message! – kkm inactive - support strike Jan 12 '20 at 04:06
2

Nice simple solution. Caveat: be careful using this as part of a cron job; process substitution is not available in some shells used by cron. – Quinn Comendant Jun 01 '20 at 17:32

score 25 · Answer 3 · answered Nov 26 '12 at 11:47

25

A naive use of process substitution seems to allow filtering of stderr separately from stdout:

:; ( echo out ; echo err >&2 ) 2> >( sed s/^/e:/ >&2 )
out
e:err

Note that stderr comes out on stderr and stdout on stdout, which we can see by wrapping the whole thing in another subshell and redirecting to files o and e

( ( echo out ; echo err >&2 ) 2> >( sed s/^/e:/ >&2 ) ) 1>o 2>e

answered Nov 26 '12 at 11:47

solidsnack

1,631
16
26

why the :; at the beginning? I tried the magic line without, and doesn't seem to make a difference. – Koshmaar Feb 08 '16 at 17:29
2

Some people use `$` as a command prompt. Then they often write shell examples like: `$ cat /var/log/syslog | fgrep ...`. However, this line is not copy-pastable because of the `$`. `:;` looks like a prompt but is basically a shell no-op; so you can select and paste the whole line safely. – solidsnack Feb 12 '16 at 09:44
27

Ok, but you could just omit the :; and the line would be also copy-pastable :) To me :; doesn't look like a prompt, it was not making the example clearer (separating commands from output) but confusing. Though I understand your point of view and let's not discuss syntax/conventions. – Koshmaar Feb 19 '16 at 16:31

score 25 · Answer 4 · edited Aug 19 '22 at 10:29

TL;DR: (bash and zsh)

$ cmd 2> >(stderr-filter >&2)

Example:

% cat /non-existant 2> >(tr o X >&2)
cat: /nXn-existant: NX such file Xr directXry
%

Many answers on the StackExchange network have the form:

cat /non-existant 3>&1 1>&2 2>&3 3>&- | sed 's/e/E/g'

This has a built-in assumption: that file descriptor 3 isn't being used for something else.

Instead, use a named file descriptor, and {ba,z}sh will allocate the next available file descriptor >= 10:

cat /non-existant {tmp}>&1 1>&2 2>&$tmp {tmp}>&- | sed 's/e/E/g'

Note that named file descriptors aren't supported by POSIX sh.

The other issue with the above is that the command cannot be piped to further commands without again swapping STDOUT and STDERR back to their original values.

To allow onward piping in POSIX sh, (and still assuming FD 3 is not it use) it gets complicated:

(cmd 2>&1 >&3 3>&- | stderr-filter >&2 3>&-) 3>&1

So, Given the assumption and gnarly syntax of this, you're likely to be better off using the simpler bash/zsh syntax shown in the TL;DR above, and explained here.

practical demonstration, grepping only stderr:

$ ls -l . noexistABC noexistXYZ
ls: cannot access 'noexistABC': No such file or directory
ls: cannot access 'noexistXYZ': No such file or directory
.:
total 4
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 bar.txt
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 foo.txt
drwxrwxr-x 2 frank frank 4096 Aug 19 12:26 someFolder


$ ( ls -l . noexistABC noexistXYZ 2>&1 >&3 3>&- | grep ABC >&2 3>&-) 3>&1
.:
ls: cannot access 'noexistABC': No such file or directory
total 4
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 bar.txt
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 foo.txt
drwxrwxr-x 2 frank frank 4096 Aug 19 12:26 someFolder

Why post the same answer twice? – Rucent88 Aug 21 '21 at 16:24 — Rucent88, Aug 21 '21 at 16:24

score 8 · Answer 5 · answered Aug 30 '12 at 10:12

I find the use of bash process substitution easier to remember and use as it reflects the original intention almost verbatim. For example:

$ cat ./p
echo stdout
echo stderr >&2
$ ./p 2> >(sed -e 's/s/S/') | sed 's/t/T/'
sTdout
STderr

uses the first sed command as a filter on stderr only and the second sed command to modify the joined output.

Note that the white space after 2> is mandatory for the command to be parsed correctly.

score 7 · Answer 6 · edited Mar 07 '17 at 07:58

7

The last part of this page of the Advanced Bash Scripting Guide is "redirecting only stderr to a pipe".

# Redirecting only stderr to a pipe.

exec 3>&1                              # Save current "value" of stdout.
ls -l 2>&1 >&3 3>&- | grep bad 3>&-    # Close fd 3 for 'grep' (but not 'ls').
#              ^^^^   ^^^^
exec 3>&-                              # Now close it for the remainder of the script.

# Thanks, S.C.

This may be what you want. If not, some other part of the ABSG should be able to help you, it is excellent.

edited Mar 07 '17 at 07:58

tripleee

175,061
34
275
318

answered Sep 01 '10 at 12:52

signine

2,985
1
18
9

We are somewhat hesitant to recommend the ABSG as a reference, as it mixes documentation, prescription, and opinion without clearly marking the differences. Some sections also have dubious content, although the one you link to seems fine. – tripleee Mar 07 '17 at 07:59

score 2 · Answer 7 · answered Sep 01 '10 at 12:56

2

Take a look at named pipes:

$ mkfifo err
$ cmd1 2>err |cat - err |cmd2

answered Sep 01 '10 at 12:56

wilhelmtell

57,473
20
96
131

won't `cat - err` break the interspersing of stdout and stderr? – Martin DeMello Sep 01 '10 at 14:15
@Martin - it depends. If cmd1, cat, **or** cmd2 buffers output, then you could see output out of sequence. – mob Sep 01 '10 at 22:31
This won't run. The shell can't finish opening `err` for writing until `cat` opens it for reading, but `cat` won't open it until it finishes reading stdin, which it can't until `cmd1` dies, but `cmd1` can't start until it finishes opening `err`. This is a deadlock. [`fifo(7)`](https://man7.org/linux/man-pages/man7/fifo.7.html) documents this: "Normally, opening the FIFO blocks until the other end is opened also." – JoL Dec 06 '21 at 22:52

Pipe only STDERR through a filter

7 Answers7

TL;DR:

Explanation

TL;DR: (bash and zsh)

Linked

Related