0

When I use a single grep command, it processes and output the data live as it comes.

Here is my simple test file test.sh:

echo a
sleep 1
echo b
sleep 1
echo ab
sleep 1
echo ba
sleep 1
echo baba

I do the following:

sh test.sh | grep a
a
ab
ba
ab
ba

all good so fa. 'a' appears immediately, then 'ab', etc.

But when I pipe multiple grep commands like that

sh ./test.sh | grep a | grep b
ab
ba
baba

I only get the output at the end, not as it comes! the terminal stays empty till the entire file is processed then outputs everything in one go.

Why is that?

How can i chain/cascade multiple greps without losing that 'process and output as it comes' property?

This is for greping and processing live huge logs with a lot of data where I only have a chance to save to disk the filtered version and not the huge raw ouput that would fill up the disk quite quickly.

Cyrus
  • 84,225
  • 14
  • 89
  • 153
MikaelW
  • 1,145
  • 4
  • 11
  • 29
  • Data **is** processed live, it's just not flushed immediately at very low volumes. If you turn on flushing, you'll actually make your total throughput worse, since content will be processed in smaller chunks rather than bigger ones. – Charles Duffy Oct 14 '17 at 13:37
  • 1
    See [BashFAQ #9](http://mywiki.wooledge.org/BashFAQ/009) for a more detailed discussion. – Charles Duffy Oct 14 '17 at 13:38
  • (BTW, if you're using `sh`, that's a different shell than bash, so you should adjust your tagging appropriately; even if your operating system uses bash to provide `/bin/sh`, it runs in compatibility mode when invoked under that name). – Charles Duffy Oct 14 '17 at 13:39

1 Answers1

2

There's an option called line-buffered:

Other Options
       --line-buffered
              Use line buffering on output.  This can cause a performance penalty.

So:

sh ./test.sh | grep --line-buffered a | grep b
Razvan
  • 2,436
  • 18
  • 23