3

How can I pipe the std-out of multiple commands to a single command? Something like:

(cat my_program/logs/log.*;tail -0f my_program/logs/log.0) | grep "filtered lines"

I want to run all the following commands on a single command-line using pipes and no redirects to a temp file (if possible). There is one small nuance that means I can't use parentheses; I want the last command to be a tail feed so I want the grep to happen after every line is received by the std-in - not wait for EOF signal.

choroba
  • 231,213
  • 25
  • 204
  • 289
ChrisN
  • 530
  • 5
  • 13

3 Answers3

8

Try using the current shell instead of a subshell:

{ cat file1; tail -f file2; } | grep something

The semi-colon before the closing brace is required.

glenn jackman
  • 238,783
  • 38
  • 220
  • 352
  • i agree its a pointless use of a new shell and may cause the output to be buffered while waiting for the EOF. but I did try that too earlier and got an endless wait for EOF so it isn't the only thing. I am surprised there isn't an easy way to do this in the command line – ChrisN Nov 03 '12 at 00:32
  • well, `tail -f` will *never* give you EOF, so why are you expecting one? `grep` will be OK operating on a stream of data with no EOF – glenn jackman Nov 03 '12 at 01:52
  • @glennjackman I used the above code to show what I needed to do, I know the command doesn't work when you use `tail -f` ,but the problem isn't the `grep` because `tail -0f my_program/logs/log.0 | grep "filtered lines"` does work. so I need some other way to concatenate `cat my_program/logs/log.*` and `tail -0f my_program/logs/log.0` so I need another way that does cause a a wait for EOF. – ChrisN Nov 03 '12 at 10:34
1

If the commands don't have to be executed in parallel then you can use cat to unify the output into a single consecutive stream, like this:

tail -0f log.0 | cat - log.* | grep "filtered lines"

The magic bit is the - parameter for cat; it tells cat to take stdin and add it to the list of inputs (the others, in this case, being logfiles) to be concatenated.

itsbruce
  • 4,825
  • 26
  • 35
  • That sadly doesn't work because it wants to put the logs after the tail feed which ends when the process is ended. The plan is to have the grep run through all the logs "cat log.*" and then poll for anything newly added to do the same to "tail -0f log.0". I would run the grep on each separately, but the grep represents just one in a list of commands piped together, so I'm concerned that anything added to this detailed log between cat and tail -0f will be lost. Thanks for the clever suggestion though Bruce – ChrisN Nov 02 '12 at 14:09
  • Did you consider using a named pipe? Run a script which contains a loop that reads from the fifo and pipes the output through your filters. Then you can toss data into the fifo as it comes. **cat log.* > /path/to/pipe** followed by **tail -0f log.0 > /path/to/pipe**, for example. – itsbruce Nov 02 '12 at 15:27
  • No I hadn't considered that. There would be a bit of tidying up to do (i.e. remove the FIFO) when the user sends a SIGINT, but I could use the trap command for that I guess. It maybe the way to go... Thanks again Bruce! – ChrisN Nov 02 '12 at 16:38
0

Okay I have a messy solution (it's a script - sadly this would be unpractical for the command line):

#!/bin/bash  

ProcessFIFO(){
    while [[ -p /tmp/logfifo ]]; do
    grep "filtered lines" < /tmp/logfifo
    done
}
OnExit(){
    rm -f /tmp/logfifo
    echo 'temp files deleted'
    exit
}
    # Create FIFO
    mkfifo /tmp/logfifo
    trap OnExit SIGINT SIGTERM
    ProcessFIFO &

    cat log.* > /tmp/logfifo
    tail -0f log.0 > /tmp/logfifo

I have used to FIFO method suggested by @itsbruce and I've also had to use a while loop to stop the grep from ending at the EOF signal: 

while [[ -p /tmp/logfifo ]]

I have also included a trap to delete the FIFO: 

trap OnExit SIGINT SIGTERM
ChrisN
  • 530
  • 5
  • 13