Write STDOUT & STDERR to a logfile, also write STDERR to screen

Question

I would like to run several commands, and capture all output to a logfile. I also want to print any errors to the screen (or optionally mail the output to someone).

Here's an example. The following command will run three commands, and will write all output (STDOUT and STDERR) into a single logfile.

{ command1 && command2 && command3 ; } > logfile.log 2>&1

Here is what I want to do with the output of these commands:

• STDERR and STDOUT for all commands goes to a logfile, in case I need it later--- I usually won't look in here unless there are problems.
• Print STDERR to the screen (or optionally, pipe to /bin/mail), so that any error stands out and doesn't get ignored.

• It would be nice if the return codes were still usable, so that I could do some error handling. Maybe I want to send email if there was an error, like this:

  { command1 && command2 && command3 ; } > logfile.log 2>&1 || mailx -s "There was an error" stefanl@example.org

The problem I run into is that STDERR loses context during I/O redirection. A '2>&1' will convert STDERR into STDOUT, and therefore I cannot view errors if I do 2> error.log

Here are a couple juicier examples. Let's pretend that I am running some familiar build commands, but I don't want the entire build to stop just because of one error so I use the '--keep-going' flag.

{ ./configure && make --keep-going && make install ; } > build.log 2>&1

Or, here's a simple (And perhaps sloppy) build and deploy script, which will keep going in the event of an error.

{ ./configure && make --keep-going && make install && rsync -av --keep-going /foo devhost:/foo} > build-and-deploy.log 2>&1

I think what I want involves some sort of Bash I/O Redirection, but I can't figure this out.

Answer 1

(./doit >> log) 2>&1 | tee -a log

This will take stdout and append it to log file.

The stderr will then get converted to stdout which is piped to tee which appends it to the log (if you are have Bash 4, you can replace 2>&1 | with |&) and sends it to stdout which will either appear on the tty or can be piped to another command.

I used append mode for both so that regardless of which order the shell redirection and tee open the file, you won't blow away the original. That said, it may be possible that stderr/stdout is interleaved in an unexpected way.

Answer 2

If your system has /dev/fd/* nodes you can do it as:

( exec 5>logfile.txt ; { command1 && command2 && command3 ;} 2>&1 >&5 | tee /dev/fd/5 )

This opens file descriptor 5 to your logfile. Executes the commands with standard error directed to standard out, standard out directed to fd 5 and pipes stdout (which now contains only stderr) to tee which duplicates the output to fd 5 which is the log file.

Answer 3

Here is how to run one or more commands, capturing the standard output and error, in the order in which they are generated, to a logfile, and displaying only the standard error on any terminal screen you like. Works in bash on linux. Probably works in most other environments. I will use an example to show how it's done.

Preliminaries:

Open two windows (shells, tmux sessions, whatever)

I will demonstrate with some test files, so create the test files:

touch /tmp/foo /tmp/foo1 /tmp/foo2

in window1:

mkfifo /tmp/fifo

0</tmp/fifo cat - >/tmp/logfile

Then, in window2:

(ls -l /tmp/foo /tmp/nofile /tmp/foo1 /tmp/nofile /tmp/nofile; echo successful test; ls /tmp/nofile1111) 2>&1 1>/tmp/fifo | tee /tmp/fifo 1>/dev/pts/2

Where you replace /dev/pts/2 with whatever tty you want the stderr to display.

The reason for the various successful and unsuccessful commands in the subshell is simply to generate a mingled stream of output and error messages, so that you can verify the correct ordering in the log file. Once you understand how it works, replace the “ls” and “echo” commands with scripts or commands of your choosing.

With this method, the ordering of output and error is preserved, the syntax is simple and clean, and there is only a single reference to the output file. Plus there is flexiblity in putting the extra copy of stderr wherever you want.

Answer 4

add this at the beginning of your script

#!/bin/bash
set -e 
outfile=logfile

exec > >(cat >> $outfile)
exec 2> >(tee -a $outfile >&2)

# write your code here

STDOUT and STDERR will be written to $outfile, only STDERR will be seen on the console

Answer 5

Try:

command 2>&1 | tee output.txt

Additionally, you can direct stdout and stderr to different places:

command > stdout.txt >& stderr.txt

command > stdout.txt |& program_for_stderr

So some combination of the above should work for you -- e.g. you could save stdout to a file, and stderr to both a file and piping to another program (with tee).