Using variables on grep –q doesn’t produce founds

Question

I need to extract entries from a log file and put them on an errors file.

I don't want to duplicate the entries on the errors file every time that the script is run, so I create this:

grep $1 $2 | while read -r line ; do
    echo "$line"
    if [ ! -z "$line" ]
    then
            echo "Line is NOT empty"
            if grep -q "$line" $3; then
                    echo "Line NOT added"
            else
                    echo $line >> $3
                    echo "Line added"
            fi
    fi
done

And is run using:

./log_monitor.sh ERROR logfile.log errors.txt

The first time that the script runs it finds the entries, and create the errors file (there are no errors file before).

The next time, this line never found the recently added lines to the errors file,

if grep -q "$line" $3;

therefore, the script adds the same entries to the errors file.

Any ideas of why this is happening?

BTW, `set -x` is your friend -- and your missing quotes are liable to be consequential. If your lines have a tab or two spaces at any point, `echo $line` will change them to a single space, whereas `echo "$line"` will represent them faithfully. — Charles Duffy, Mar 01 '17 at 18:03

that other guy · Accepted Answer · 2017-03-01T17:13:55.103

This most likely happens because you are not searching for the line itself, but treating the line as regex. Let's say you have this file:

$ cat file
[ERROR] This is a test
O This is a test

and you try to find the first line:

$ grep "[ERROR] This is a test" file
O This is a test

As you can see, it does not match the line we're looking for (causing duplicates) and does match a different line (causing dropped entries). You can instead use -F -x to search for literal strings matching the full line:

$ grep -F -x "[ERROR] This is a test" file
[ERROR] This is a test

Applying this to your script:

grep $1 $2 | while read -r line ; do
    echo "$line"
    if [ ! -z "$line" ]
    then
            echo "Line is NOT empty"
            if grep -F -x -q "$line" $3; then
                    echo "Line NOT added"
            else
                    echo $line >> $3
                    echo "Line added"
            fi
    fi
done

And here with additional fixes and cleanup:

grep -e "$1" -- "$2" | while IFS= read -r line ; do
    printf '%s\n' "$line"
    if [ "$line" ]
    then
            echo "Line is NOT empty"
            if grep -Fxq -e "$line" -- "$3"; then
                    echo "Line NOT added"
            else
                    printf '%s\n' "$line" >> "$3"
                    echo "Line added"
            fi
    fi
done

PS: this could be shorter, faster and have a better time complexity with a snippet of awk.

Thank you, your fixed and cleanup code was the winner. It seems that the problem was when saving the line into the errors.txt; echo does it in a different way than printf. The first example (using my echo) didn't work, but using printf and \n worked perfectly. Thank you again! — Santiago Martinez, Mar 01 '17 at 17:57
@SantiagoMartinez, *nod* -- see [the POSIX standard for `echo`](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html), which describes in detail how implementations vary and explicitly recommends using `printf` instead for new development. — Charles Duffy, Mar 01 '17 at 18:02
The fix wasn't using `printf` as much as quoting the variable. If your lines contain glob characters (`[]*?`), tabs, multiple spaces in a row or similar, this would cause [echo to write something else](http://stackoverflow.com/questions/29378566/i-just-assigned-a-variable-but-echo-variable-shows-something-else). — that other guy, Mar 01 '17 at 18:49

agc · Answer 2 · 2017-03-01T18:12:43.010

0

There's no need to check for blank lines; the first grep only checks lines with the word "ERROR", which cannot be blank.

If you can do without the diagnostic echo messages, pretty much the whole of that code boils down what might be done using two greps, a bash process substitution, sponge, and touch for the first-run case:

[ ! -f $3 ] && touch $3 ; grep -vf $3 <(grep $1 $2) | sponge -a $3

edited Mar 01 '17 at 18:12

answered Mar 01 '17 at 17:23

agc

7,973
2
29
50

Using variables on grep –q doesn’t produce founds

2 Answers2