awk and data stream through pipe not working

Question

I am facing an issue today with awk command and data stream using fifo under ubuntu 14.04.

I have a writer.sh providing data

#!/bin/bash

PIPE_NAME="pipe.fifo"
DATA="dataA,dataB"

# create fifo if doesn't exist
if ! [ -p "$PIPE_NAME" ]; then
    mkfifo $PIPE_NAME
fi
# keep fifo openned
sleep 10000 > $PIPE_NAME & 

while [[ true ]];
do
    echo $DATA > $PIPE_NAME
    sleep 1
done

which is working fine.

On another shell, if i write the command:

cat pipe.fifo|grep data|awk -F, '{print $1}'

I don't get anything.

I've tried creating a dataList.txt file with the following content:

dataA,dataB
dataA,dataB
dataA,dataB

The following command:

cat dataList.txt|grep data|awk -F, '{print $1}'

produces the expected result:

dataA
dataA
dataA

Note: The grep command doesn't have any interest here but in my application, it is useful to get only data matching a certain pattern.

Do you have any idea of what I misunderstand ?

pipe.fifo is the fifo created in the writer.sh script with mkfifo. The only data in this fifo are the ones generated by writer.sh script with line "echo $DATA > $PIPE_NAME" — Martin, Mar 02 '17 at 15:51
As long as the input stream is open, grep may buffer its output when not writing to the terminal. See here: http://stackoverflow.com/questions/7161821/how-to-grep-a-continuous-stream — jas, Mar 02 '17 at 16:30
Thanks for the suggestion, however, using pipe.fifo or dataList.txt has no difference on the grep output. In other words, both cat dataList.txt|grep data and cat pipe.fifo|grep data produce the same ouput. The difference is if awk is added through a pipe. — Martin, Mar 02 '17 at 16:38
Just to be clear, you tried the line-buffered option: `grep --line-buffered data` ? — jas, Mar 02 '17 at 16:52
That's not a great way to keep the pipe open. Perhaps your want `while :; do echo $DATA; sleep 1; done > "$PIPE_NAME"` instead? — chepner, Mar 02 '17 at 17:04
I've tried "grep --line-buffered data" without success (same behaviour as what I explain above). I also tried your solution chepner but without a giant sleep in the fifo, i can't keep it opened :/ If you have any other idea, you're very welcome to share.. — Martin, Mar 02 '17 at 17:21
`grep` isn't blocking on its input, `awk` is, because `grep` is buffering its output. `grep` won't exit (and flush its buffer) until `writer.sh` closes its end of the pipe. — chepner, Mar 02 '17 at 17:31

score 2 · Accepted Answer · answered Mar 02 '17 at 17:37

The first write to the pipe in writer.sh will block until something reads from the pipe. I'd suggest writing writer.sh like this:

#!/bin/bash

pipe_name="pipe.fifo"
data="dataA,dataB"

[ -p "$pipe_name" ] || mkfifo "$pipe_name"

while :; do
    echo "$data"
    sleep 1
done > "$pipe_name"

Here, the pipe will stay open until your while loop exits.

If grep is buffering its output, then it is continuously reading from the pipe as data arrives, but it won't output anything to the pipe to awk until that buffer is full or until it exists. Since writer.sh is in an infinite loop, you just have to wait until the buffer is full.

Since the combination of grep | awk is somewhat of an anti-pattern (since awk can do its own filtering), you can replace your reader with

awk -F, '/data/ {print $1}' pipe.fifo

which, since it is writing to the terminal, will not buffer its output.

With your solution, awk is giving me a result only if I stop the while loop in the writer. I would rather have a "real time" solution but i'm starting to think using a combination of pipes/commands is simply not the best way. And i'll start working on a specific script for that. Thanks for the solution anyway, it's still simpler — Martin, Mar 03 '17 at 08:22

awk and data stream through pipe not working

1 Answers1