1

As in the title, how can I compare stdout & stderr of a program using diff on a terminal (without using temporary files or named pipes)?

Put more generally, I would like to do something like this

             (stdout) | <program B>
            /                       \
<program A>                           <diff> - terminal
            \                       /
             (stderr) | <program C>

I thought that I could achieve the task in a similar manner to this. It suggests several options, which include

1. using temp files

 foo | bar > file1.txt && baz | quux > file2.txt && diff file1.txt file2.txt

I don't want to use temp files :(

2. using process substitution

diff <(foo | bar) <(baz | quux)

It compares two streams (both stdout) from different commands using the file descriptor trick. What I want is to compare two streams of a single command, which could not be done with the trick...

3. using named pipe

mkfifo file1_pipe.txt
foo|bar > file1_pipe.txt && baz | quux | diff file1_pipe.txt - && rm file1_pipe.txt

I want named pipe neither :(

EDIT: Sorry for being a bit unclear on my intention of the question. My interest was to see the capability of bash (or other shells) to handle complex steam redirections.

bivoje
  • 23
  • 1
  • 7
  • @bivoje – Do you have any requirement regarding the handling of STDERR from `bar` and `quux`? – Armali Mar 14 '23 at 10:40
  • 1
    `without using temporary file or named pipe` Why not just use the temporary file or a pipe? `I don't want named pipe neither` Why not? You can spend your time on it, but this is _shell_, not a programming language, using temporary file and a pipe _are_ the tools you use in shell. Have you considered using python or perl? – KamilCuk Mar 14 '23 at 14:03
  • 1
    Can you say something more specific than "don't want to" to make it clear why some things are allowed and others aren't? ("Data is sensitive and must not touch a file" is one example of a practical rationale, but that one doesn't rule out named pipes, since the data passing through them doesn't ever go to disk). Without a practical evaluation criteria, it's hard to tell whether a technique you haven't foreseen would be considered allowable. – Charles Duffy Mar 14 '23 at 14:15
  • 1
    @Fravadona, I don't see why you'd need two temporary files. One named FIFO and one anonymous FIFO should be enough, as long as the `diff` implementation in use is single-pass and doesn't need to rewind. `mkfifo stderr; diff <(foo 2>stderr) stderr` and there you are. (Not an answer since the OP isn't allowing named FIFOs; I'm pondering whether bash's `coproc` implementation requires stdin to be a read-only FD, or if we could smuggle something writable through that way). – Charles Duffy Mar 14 '23 at 14:18

3 Answers3

3

If you have a sufficiently new version of Bash (at least 4.3 (2014)) and an operating system that supports /proc/PID/fd (definitely Linux, maybe Solaris, possibly others) then you can use coprocesses to do what you want:

#! /bin/bash -p

# Start coprocesses to process command standard output and standard error
coproc outproc { cat; }
# (Redirect stderr to suppress a spurious warning about an existing coprocess)
{ coproc errproc { cat; }; } 2>/dev/null

# Run 'diff' in the background, taking inputs from the coprocesses
diff "/proc/$$/fd/${outproc[0]}" "/proc/$$/fd/${errproc[0]}" &

# Run the program to completion, redirecting standard output and standard
# error to coprocesses
prog >&"${outproc[1]}" 2>&"${errproc[1]}"

# Close the pipes to the coprocesses, causing them to exit
exec {outproc[1]}>&-
exec {errproc[1]}>&-

# Wait for 'diff' and the coprocesses to exit
wait
  • I tested the code successfully (with a predefined prog function) with Bash 5.1.16 on Ubuntu 22.04.
  • The code failed (hang) with Bash 4.4.12 on Cygwin. I don't know if that is because of the Bash version or issues with the Linux emulation in Cygwin.
  • Replace the cat in the coprocess bodies with code to do processing of the stdout and stderr data (e.g. program_B and program_C).
  • See the (excellent) accepted answer to How do you use the command coproc in various shells? for detailed information about how to use coprocesses.
  • This technique will work with arbitrary output data. Techniques that involve storing data in Bash variables, or using NUL characters to separate stdout and stderr data, won't work with data that contains NUL characters.
pjh
  • 6,388
  • 2
  • 16
  • 17
  • 1
    Very nice! I started playing around in this direction but didn't get something complete enough to work before needing to get back to work. Congrats for pulling it all together. :) – Charles Duffy Mar 15 '23 at 05:13
  • 2
    Thanks! Your answer fulfilled my curiosity. Also, it was a good read you gave too! – bivoje Mar 15 '23 at 12:28
1

If the stdout and stderr outputs are small enough to be stored in Bash variables, you could use one of the techniques described in answers to Capture stdout and stderr into different variables to capture them into (say) out and err variables and then run

diff <(printf '%s\n' "$out") <(printf '%s\n' "$err")
pjh
  • 6,388
  • 2
  • 16
  • 17
  • 1
    Capturing them in variables is equivalent to capturing them in a file somehow. – hacker315 Mar 14 '23 at 14:09
  • 1
    @hacker315, several of the answers to [Capture stdout and stderr into different variables](https://stackoverflow.com/q/11027679/4154375) demonstrate ways to capture to variables without using the file system at all. – pjh Mar 14 '23 at 14:40
0

You can serialize messages from two streams into one stream using some protocol, and then deserialize the data later into two variables. For example, using a prefix on each line to mark the line and then recognizing and removing the prefix

output=$(
    exec 3>&1 1>&2
    program 2> >(sed 's/^/E/' >&3) | sed 's/^/O/' >&3
)
diff <(sed -n 's/^E//p' <<<"$output") <(sed -n 's/^O//p' <<<"$output")

Or for example, store it all in a variable and serialize as a bash source-able format:

output=$(
    exec 3>&1 1>&2
    program 2> >(
        stderr=$(cat)
        flock 1
        declare -p stderr >&3
    ) |
        (
            stdout=$(cat)
            flock 1
            declare -p stdout >&3
        )
)
eval "$output"
diff <(cat <<<"$stdout") - <<<"$stderr"
KamilCuk
  • 120,984
  • 8
  • 59
  • 111