Bash Script Command Redirection/Reuse -- How to Accomplish Multidirectional Layered Redirection?

Question

I'm struggling to understand command redirection/reuse...

I understand there's the <(...) <(...) methodology and the $( ... && ... ) techniques for combining output. But I don't really fully understand what the difference is (I realize that the (...) dumps you inside a new shell, causing you to potentially jump dirs and lose any vars you haven't exported, but I'm unsure of how it effects the general redirection scheme) and I'm still quite confused as to how to do one-to-many redirection after skimming over the examples and instructions in:
Advanced Bash Scripting Guide 1
Advanced Bash Scripting Guide 2

My own attempts to play around with it have mostly resulted in "ambiguous redirect" errors.

For example, let's say I want to do a one liner given by the pseudocode below

    CMD 1 && CMD 2 && CMD 3 --> (1)  
    CMD 4 (1) --> (2)
    CMD 5 (1) --> CMD 6 --> (3)
    CMD 7 (2) && CMD 8 (3) --> (4)
    CMD 9 (2) --> (5)
    CMD 10 (2) (3) -->(6)
    VAR= echo (4) && echo (5) && echo (6)

Or as a process diagram

CMD 1 +CMD 2 && CMD 3 
    |\
    | ---> CMD 5 ------> CMD 6-----\
    V                     /         V
    CMD 4 ----------------u--------> CMD 10
     | \                  V           /
     |   -------->CMD 7 + CMD 8      /
     V             |                /
    CMD 9          |               /
      \            |              /
       \           V             /
        --------> VAR <----------

Where outputs are designated as -->; storage for reuse in another op is given by -->(#); and combination operations are given by &&.

I currently have no idea how to do this in a single line without redundant code.

I want to truly master command redirection so I can make some powerful one-liners.

Hopefully that's clear enough... I could come up with proof of concept examples for the commands, if you need them, but the pseudocode should give you the idea.

Answer 1

There's some kind of beginning of answer to your problem in this post: How can I send the stdout of one process to multiple processes using (preferably unnamed) pipes in Unix (or Windows)?

Using tee and >(command). But your example is rather sophisticated, and it's hard to imagine a one-line solution, without temp storage (var or file), by only combining commands (the Unix way, yes!).

I mean, if you agree to have several commands launched sequentially, it may be easier (but it won't be a one-liner any longer)...

Even writing this kind of expression with a more complex language (say python) would be complicated.

Answer 2

In reply to your comment:

sort <(cd $CURR_DIR && find . -type f -ctime $FTIME) \ 
    <(cd $CURR_DIR && find . -type f -atime $FTIME) \ 
    <(cd $CURR_DIR && find . -type f -mtime $FTIME) | uniq

can be written as (which I believe is clearer)

(find . -type f -ctime $FTIME && find . -type f -atime $FTIME \
    && find . -type f -mtime $FTIME) | sort | uniq

Given three programs one which produces "a" or "z" as output. Produce a string that contains the sorted output and also the unique output as a one liner:

mkfifo named_pipe{1,2,3}; (echo z ; echo a ; echo z) > named_pipe1 & \
    tee < named_pipe1 >(sort > named_pipe2) | sort | uniq > named_pipe3 & \
    output=$(echo sorted `cat named_pipe2`; echo unique `cat named_pipe3`); \
    rm named_pipe{1,2,3}

Produces sorted a z z unique a z

One thing you may notice about this solution is that its been split up so that each grouping of commands has its own line. I suppose what I'm getting at here is that one liners can be cool, but often clarity is better.

The mechanism by how this works is use of a program called tee and named pipes. A named pipe is exactly like an anonymous pipe eg. cat words.txt | gzip, except that it can be referenced from the file system (but no actual data is written to the file system). Note that writing to a named pipe will block until another process is reading from the named pipe. I've been gratuitous with my use of pipes here, just so you can get a feel of how to use them.

Tee, as noted by others, can replicate input to several outputs.

Answer 3

You have several different problems to solve.

1. You have one output that you need as input for several other commands. You can solve this with tee >(cmd1) | cmd2. See this answer: How can I send the stdout of one process to multiple processes using (preferably unnamed) pipes in Unix (or Windows)?

   An alternative is to create a file. I usually prefer the file approach because it allows to debug the final script. To avoid leaving a mess, create a temporary work directory which you delete with trap "..." EXIT

2. You need to combine the outputs of several commands as the input to a single command. Use sub shells here: ( CMD 1 ; CMD 2 ) | CMD 3 combines the output of 1 and 2 as input for 3.

3. You need a mix of the two above. You can create additional file descriptors but each can be read only once. Use cat to combine them and tee to create copies.

This should all be possible but there is a drawback: If something doesn't work or goes wrong, you will never be able to find the bug.

Therefore: Create a work directory and use files. If you put this into /tmp, the data won't be written to disk (on modern Linux systems, /tmp is a ram disk), this will behave like pipes except when you have huge amounts of data plus you will be able to maintain the script.