192

I have a number of lines retrieved from a file after running the grep command as follows:

var=`grep xyz abc.txt`

Let’s say I got 10 lines which consists of xyz as a result.

Now I need to process each line I got as a result of the grep command. How do I proceed for this?

oguz ismail
  • 1
  • 16
  • 47
  • 69
XYZ_Linux
  • 3,247
  • 5
  • 31
  • 34
  • 6
    None of the answers here mention the power of `grep -o` for this sort of thing. The `-o` flag will give back _only_ the text that matches, with one match per line of output. (It's not exhaustive, so `echo aaa |grep 'a*'` only gives you "aaa" and omits the three partial matches "", "a", and "aa") – Adam Katz Dec 14 '16 at 21:02

7 Answers7

341

One of the easy ways is not to store the output in a variable, but directly iterate over it with a while/read loop.

Something like:

grep xyz abc.txt | while read -r line ; do
    echo "Processing $line"
    # your code goes here
done

There are variations on this scheme depending on exactly what you're after.

If you need to change variables inside the loop (and have that change be visible outside of it), you can use process substitution as stated in fedorqui's answer:

while read -r line ; do
    echo "Processing $line"
    # your code goes here
done < <(grep xyz abc.txt)
Community
  • 1
  • 1
Mat
  • 202,337
  • 40
  • 393
  • 406
  • if there is no line with xyz ? – XYZ_Linux May 01 '13 at 12:24
  • Then nothing happens, the loop is not run. – Mat May 01 '13 at 12:26
  • 16
    The problem with this method is that (because of the pipe) everything inside the loop is in a subshell, so setting variables defined outside the loop during the loop does not make their values available after the loop! – David Doria Jan 24 '14 at 15:50
  • 4
    @David: provided an alternative to address your concern. (fedorqui had already addressed it too.) – Mat Jan 24 '14 at 15:52
  • For commands whose last line of output is not terminated with a newline, you'd need: `while read p || [[ -n $p ]]; do ...` (borrowed from http://stackoverflow.com/questions/1521462/looping-through-the-content-of-a-file-in-bash#comment31060693_1521498) – Ohad Schneider Feb 22 '16 at 09:59
  • it seems that both of these solutions disable color in whatever is run inside the loop.. is this solvable somehow? – davidhq Mar 30 '16 at 19:34
  • I also added `--line-buffered` to my grep command on MacOSX, as this version of grep seems to buffer it's output, meaning that I did not see the output streaming – russholio May 26 '20 at 03:53
27

You can do the following while read loop, that will be fed by the result of the grep command using the so called process substitution:

while IFS= read -r result
do
    #whatever with value $result
done < <(grep "xyz" abc.txt)

This way, you don't have to store the result in a variable, but directly "inject" its output to the loop.

Note the usage of IFS= and read -r according to the recommendations in BashFAQ/001: How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?:

The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters). Without this option, any unescaped backslashes in the input will be discarded. You should almost always use the -r option with read.

In the scenario above IFS= prevents trimming of leading and trailing whitespace. Remove it if you want this effect.

Regarding the process substitution, it is explained in the bash hackers page:

Process substitution is a form of redirection where the input or output of a process (some sequence of commands) appear as a temporary file.

fedorqui
  • 275,237
  • 103
  • 548
  • 598
  • OK, I striked the `for` version. Tried to do a loop on `"${$(grep xyz abc.txt)[@]}"` as in http://stackoverflow.com/a/14588210/1983854 but could not. So I just leave the first version. – fedorqui May 01 '13 at 15:36
  • 1
    You can't apply parameter expansion to a command substitution (unless you're using `zsh`, where that kind of nesting probably works). – chepner May 01 '13 at 15:39
  • 1
    One possible problem with this idiom is that if anything inside the loop tries to read from standard input, it'll get part of the file. To avoid this possibility, I like to send the file through file descriptor 3 rather than stdin. Just use `while IFS= read -r result <&3` and `done 3< <(grep ...` – Gordon Davisson Jun 13 '19 at 20:46
  • I know the original question asked for bash, but for future readers: this is not POSIX-compliant. shellcheck(SC2039): In POSIX sh, process substitution is undefined. – zypA13510 Apr 09 '21 at 11:03
23

For those looking for a one-liner:

grep xyz abc.txt | while read -r line; do echo "Processing $line"; done
laurent
  • 88,262
  • 77
  • 290
  • 428
9

I would suggest using awk instead of grep + something else here.

awk '$0~/xyz/{ //your code goes here}' abc.txt

Julien Grenier
  • 3,364
  • 2
  • 30
  • 43
2

Without any iteration with the --line-buffered grep option:

your_command | grep --line-buffered "your search"

Real life exemple with a Symfony PHP Framework router debug command ouput, to grep all "api" related routes:

php bin/console d:r | grep --line-buffered "api"
Nicolas Bonnici
  • 423
  • 4
  • 11
  • The only solution that works if your_command is long-running (such as `tail -f some.log`, in my case)... – Izkata Sep 27 '19 at 18:28
1

Iterate over the grep results with a while/read loop. Like:

grep pattern filename.txt | while read -r line ; do
    echo "Matched Line:  $line"
    # your code goes here
done
Abhi km
  • 159
  • 1
  • 5
0

Often the order of the processing does not matter. GNU Parallel is made for this situation:

grep xyz abc.txt | parallel echo do stuff to {}

If you processing is more like:

grep xyz abc.txt | myprogram_reading_from_stdin

and myprogram is slow then you can run:

grep xyz abc.txt | parallel --pipe myprogram_reading_from_stdin
Ole Tange
  • 31,768
  • 5
  • 86
  • 104