Bash not waiting for full results after pipe but immediate action on direct output

Question

When using grep -r "something" ., it will print to output as something gets found.

Whenever using a pipe afterwards to postprocess, it will wait though until having received all the output.

Is there some way / syntax for piping per line as it gets available?

Don't bother with improving the example, it really is just an example:

grep -r "something" . | grep -v "somethingelse"  # excluding somethingelse

The result will only be there after everything has been searched (because of the pipe |), but I'm looking for syntax which would respond on each line from grep -r "something" . as the "line becomes available".

It's possible to pipe to a program that waits for input and handles anything that reaches it, but using the pipe will only send data to it as it is done completely?

Does anyone have an idea?

You can use `unbuffer` command if you have `expect` package installed. Similar issue was discussed [here](http://stackoverflow.com/questions/3465619/how-to-make-output-of-any-shell-command-unbuffered) — Balaji Sukumaran, Dec 19 '15 at 21:34
@BalajiSukumaran Indeed that's a duplicate, didn't know the terminology to find it. Thanks! — PascalVKooten, Dec 19 '15 at 21:35

user2683246 · Accepted Answer · 2015-12-19T21:08:45.033

3

With grep you can use --line-buffered option. If program does not have a similar option, you can use stdbuf program, e.g.:

tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq

In this example stdbuf runs cut program with its output stream modified to be line buffered.

edited Dec 19 '15 at 21:08

answered Dec 19 '15 at 21:06

user2683246

3,399
29
31

Doesn't the pipe make us wait for the tailing to stop (which it wouldn't) before showing results? – PascalVKooten Dec 19 '15 at 21:08
it is not pipe that makes us wait. it is buffering mode of output stream attached to that pipe that does. – user2683246 Dec 19 '15 at 21:13
Why does `grep -rs "a" / | /usr/local/opt/coreutils/libexec/gnubin/stdbuf -oL echo "$(date)` yield only 1 line of output? – PascalVKooten Dec 19 '15 at 21:23
because `echo` does not read stdin at all, so it useless after pipe, unless you pass it arguments from stdin with `xargs`, e.g. `cat somefile | xargs echo $(date)` – user2683246 Dec 19 '15 at 21:45
`grep -rsZ a / | xargs -0L1 echo $(date)` – user2683246 Dec 21 '15 at 17:36

score 1 · Answer 2 · answered Dec 19 '15 at 20:59

1

Use grep's --line-buffered option:

--line-buffered
         Force output to be line buffered.  By default, output is line
         buffered when standard output is a terminal and block buffered
         otherwise.

answered Dec 19 '15 at 20:59

chepner

497,756
71
530
681

1

I'm looking for a more general option than only in `grep`, though it is a nice find. – PascalVKooten Dec 19 '15 at 21:07

Bash not waiting for full results after pipe but immediate action on direct output

2 Answers2