Speed up bash filter function to run commands consecutively instead of per line

Question

I have written the following filter as a function in my ~/.bash_profile:

hilite() {
 export REGEX_SED=$(echo $1 | sed "s/[|()]/\\\&/g")
 while read line
 do
  echo $line | egrep "$1" | sed "s/$REGEX_SED/\x1b[7m&\x1b[0m/g"
 done
 exit 0
}

to find lines of anything piped into it matching a regular expression, and highlight matches using ANSI escape codes on a VT100-compatible terminal.

For example, the following finds and highlights the strings bin, U or 1 which are whole words in the last 10 lines of /etc/passwd:

tail /etc/passwd | hilite "\b(bin|[U1])\b"

However, the script runs very slowly as each line forks an echo, egrep and sed.

In this case, it would be more efficient to do egrep on the entire input, and then run sed on its output.

How can I modify my function to do this? I would prefer to not create any temporary files if possible.

P.S. Is there another way to find and highlight lines in a similar way?

`exit 0` is probably not what you want. If you inadvertantly type `tail file; hilite foo`, your shell will terminate! Use `:`, `true`, `return 0`, or just delete the `exit 0` line completely. — William Pursell, Oct 31 '12 at 12:30
@WilliamPursell: You're right, thanks! Guess http://stackoverflow.com/questions/5104426/while-loop-in-bash-script should be fixed too... — Gnubie, Nov 01 '12 at 19:40

doubleDown · Answer 1 · 2012-11-02T05:29:28.493

2

I think you can replace the whole while loop with simply

sed -n "s/$REGEX_SED/\x1b[7m&\x1b[0m/gp"

because sed can read from stdin line-by-line so you don't need read

I'm not sure if running egrep and piping to sed is faster than using sed alone, but you can always compare using time.

Edit: added -n and p to sed to print only highlighted lines.

edited Nov 02 '12 at 05:29

answered Oct 31 '12 at 14:34

doubleDown

8,048
1
32
48

Fixed that by simply using `-n` option and setting `p` flag for `s` command – doubleDown Nov 02 '12 at 05:30

score 2 · Answer 2 · answered Oct 31 '12 at 15:10

2

If your egrep supports --color, just put this in .bash_profile:

  hilite() { command egrep --color=auto "$@"; }

(Personally, I would name the function egrep; hence the usage of command).

answered Oct 31 '12 at 15:10

William Pursell

204,365
48
270
300

Though I prefer reverse video, that's wonderful! The colour can also be changed using GREP_COLOR or GREP_COLORS: http://www.gnu.org/software/grep/manual/html_node/Environment-Variables.html . For a single command: http://stackoverflow.com/questions/3420845/in-bash-how-to-write-alias-or-function-for-export-grep-color-132-grep-co . – Gnubie Nov 01 '12 at 19:55

evil otto · Accepted Answer · 2012-11-02T01:10:12.597

2

sed can do a bit of grepping itself: if you give it the -n flag (or #n instruction in a script) it won't echo any output unless asked. So

while read line
 do
  echo $line | egrep "$1" | sed "s/$REGEX_SED/\x1b[7m&\x1b[0m/g"
 done

could be simplified to

sed -n "s/$REGEX_SED/\x1b[7m&\x1b[0m/gp"

EDIT: Here's the whole function:

hilite() { 
    REGEX_SED=$(echo $1 | sed "s/[|()]/\\\&/g");
    sed -n "s/$REGEX_SED/\x1b[7m&\x1b[0m/gp"
}

That's all there is to it - no while loop, reading, grepping, etc.

edited Nov 02 '12 at 01:10

answered Oct 31 '12 at 19:10

evil otto

10,348
25
38

1

No, the egrep is NOT needed. The whole point of using `-n` and the `p` flag is to not print anything out unless it matches. Which is exactly the same thing that grep does. `sed -n '/foo/p'` is just another way to write `grep foo`. – evil otto Nov 02 '12 at 01:04

score 1 · Answer 4 · answered Oct 31 '12 at 12:28

1

Well, you could simply do this:

egrep "$1" $line | sed "s/$REGEX_SED/\x1b[7m&\x1b[0m/g"

But I'm not sure that it'll be that much faster ; )

answered Oct 31 '12 at 12:28

tamasgal

24,826
18
96
135

1

I don't know, but `echo` is probably much lighter on resources compared to `egrep` and `sed`. Anyway won't `egrep` treat `$line` as a file to be parsed in the command line you wrote? – doubleDown Oct 31 '12 at 14:38
Sorry, the syntax is egrep as doubleDown commented. – Gnubie Nov 01 '12 at 19:49

score 0 · Answer 5 · answered Nov 01 '12 at 20:05

Just for the record, this is a method using a temporary file:

hilite() {
 export REGEX_SED=$(echo $1 | sed "s/[|()]/\\\&/g")
 export FILE=$2
 if [ -z "$FILE" ]
 then
  export FILE=~/tmp
  echo -n > $FILE
  while read line
  do
   echo $line >> $FILE
  done
 fi
 egrep "$1" $FILE | sed "s/$REGEX_SED/\x1b[7m&\x1b[0m/g"
 return $?
}

which also takes a file/pathname as the second argument, for case like

cat /etc/passwd | hilite "\b(bin|[U1])\b"

Speed up bash filter function to run commands consecutively instead of per line

5 Answers5