297

I am passing a list of regex patterns to grep to check against a syslog file. They are usually matching an IP address and log entry;

grep "1\.2\.3\.4.*Has exploded" syslog.log

It's just a list of patterns like the "1\.2\.3\.4.*Has exploded" part I am passing, in a loop, so I can't pass "-v", for example.

I am confused trying to do the inverse of the above, and not match lines with a certain IP address and error so "!1.2.3.4.*Has exploded" will match syslog lines for anything other than 1.2.3.4 telling me it has exploded. I must be able to include an IP address to not match.

I have seen various similar posts on Stack Overflow. However, they use regex patterns that I can't seem to get to work with grep. What would be a working example for grep?

This is happening in a script like this;

patterns[1]="1\.2\.3\.4.*Has exploded"
patterns[2]="5\.6\.7\.8.*Has died"
patterns[3]="\!9\.10\.11\.12.*Has exploded"

for i in {1..3}
do
  grep "${patterns[$i]}" logfile.log
done
Benjamin Loison
  • 3,782
  • 4
  • 16
  • 33
jwbensley
  • 10,534
  • 19
  • 75
  • 93
  • Do you mean you *sometimes* want to match a pattern, but other times want to match everything *except* a certain pattern? (this seems like an odd requirement, but whatever). In that case, why don't you iterate over two different lists of patterns? – beerbajay May 02 '12 at 10:42
  • Well I'm not very knowledgeable about regex; I don't want to grep for "Has Exploded" because I don't want to know this about every logging device, so can I somehow grep for "Has Exploded" and !9.10.11.12 in one statement? – jwbensley May 02 '12 at 12:06
  • If you absolutely must do it in one statement, negative lookbehinds are the way to go, as Neil suggests. See my comment there. – beerbajay May 02 '12 at 12:57
  • Use PCRE-style regex matching, and a negative lookahead assertion, as per @Neil 's answer: `patterns[3]="\!9\.10\.11\.12.*Has exploded"` changes to `patterns[3]="(?<!9\.10\.11\.12).*Has exploded"` and `grep "${patterns[$i]}" logfile.log` changes to `grep -P "${patterns[$i]}" logfile.log` PCRE assumes more metacharacters by default, so some of the escapes may need to be removed from other matching expressions. – Codex24 Jun 05 '19 at 16:45

4 Answers4

544

grep matches, grep -v does the inverse. If you need to "match A but not B" you usually use pipes:

grep "${PATT}" file | grep -v "${NOTPATT}"
Benjamin Loison
  • 3,782
  • 4
  • 16
  • 33
beerbajay
  • 19,652
  • 6
  • 58
  • 75
  • This is going into the middle of a loop as I mentioned and I'm just passing the PATTERN to grep so I can't use "-v" as I mentioned. I'm just looping round a list of PATTERNs and passing to grep. – jwbensley May 02 '12 at 10:21
  • 3
    You can indeed use `-v` and you can use it in a loop. Perhaps you need to be more specific about your limitations, or perhaps you have a misconception about how your script should work. Try posting some code. – beerbajay May 02 '12 at 10:25
  • Thanks beerbajay, I have added a code snipped to the original post to give some context. Do you see what I mean now? – jwbensley May 02 '12 at 10:36
  • This answer isn't completely correct but you were pretty much write beerbajay, I needed to rethink the loop and in use -v in the end. Thanks for the pointer ;) – jwbensley May 03 '12 at 16:05
  • 1
    But what if A is composed of B? In other words, what if I want to match lines with **no** A **and** lines with AB ? A pipe will not work. – pawamoy Dec 02 '15 at 20:25
  • grep -v for me DOES NOT work: it still outputs the lines containing the words in the NOTPATT – Ihor B. Feb 22 '16 at 14:00
  • @IhorB. Then something else is wrong with your patterns; try testing each pattern individually with some test data to see that it gives your expected results. Perhaps you're using an unescaped special character, e.g. `.`? – beerbajay Feb 22 '16 at 23:17
  • @beerbajay, I want to get the files that contain String1, but not String2. I used the command --> grep -Hrn "String1" . | grep -v -Hrn "String2" But it still prints the files containing both the strings.. What is wrong with my command? – Bhavuk Mathur Apr 06 '16 at 06:52
  • @BhavukMathur The problem is that grep is processing the data by-line, not by file. The output of your first grep command is a bunch of text; this is sent to the second command which then filters out the **lines** of this text which match `String2`. You can do what you want with a combination of: `grep -l` and `comm`. Example: `mkdir files && cd files && echo "String1" > f1 ; echo "String2" > f2; echo -e "String1\nString2" > f3 && cd ..` then: `grep -lr String1 ./files > s1 ; grep -lr String2 ./files > s2` and finally: `comm -2 -3 s1 s2` gives: `./files/f1` – beerbajay Apr 07 '16 at 16:22
  • Same answer: https://stackoverflow.com/questions/4538253/how-can-i-exclude-one-word-with-grep/4538335#4538335 – Gabriel Staples Apr 19 '20 at 07:43
21
(?<!1\.2\.3\.4).*Has exploded

You need to run this with -P to have negative lookbehind (Perl regular expression), so the command is:

grep -P '(?<!1\.2\.3\.4).*Has exploded' test.log

Try this. It uses negative lookbehind to ignore the line if it is preceded by 1.2.3.4.

Benjamin Loison
  • 3,782
  • 4
  • 16
  • 33
Neil
  • 5,762
  • 24
  • 36
  • 3
    I'm pretty sure that `grep` doesn't support lookaround. Unless you're using Gnu `grep` and use the `--P` parameter to make it use a PCRE engine. – Tim Pietzcker May 02 '12 at 10:15
  • No, grep doesn't support this type of Regex; $grep -P (?<\!1\.2\.3\.4) test.log -bash: syntax error near unexpected token `(' – jwbensley May 02 '12 at 10:20
  • You're going to need to quote the regex if it contains characters which would be interpreted by the shell. – beerbajay May 02 '12 at 10:26
  • correct quoting: `grep -P '(?<!1\.2\.3\.4) Has exploded' test.log` Note that the lookbehind only works on the characters immediately preceding the matching part of the expression, so if there's other things between the address and message, e.g. `1.2.3.4 FOO Has exploded`, this won't work. – beerbajay May 02 '12 at 12:57
  • @TimPietzcker, very observant. I'll add that to the question. Also, please note that there is a `.*` after the negative lookbehind since his example also has it, I imagine there might be other text in between. – Neil May 02 '12 at 13:01
12

It seems no one has posted a blend of the best of all answers, regex (-E) with match inversion (-v)

grep -Ev 'pattern1|pattern2|pattern3' file

Notably, no lookarounds required, so this works if your grep version doesn't have -P available.

ZX9
  • 898
  • 2
  • 16
  • 34
2
patterns[1]="1\.2\.3\.4.*Has exploded"
patterns[2]="5\.6\.7\.8.*Has died"
patterns[3]="\!9\.10\.11\.12.*Has exploded"

for i in {1..3}
  do
grep "${patterns[$i]}" logfile.log
done

should be the the same as

egrep "(1\.2\.3\.4.*Has exploded|5\.6\.7\.8.*Has died)" logfile.log | egrep -v "9\.10\.11\.12.*Has exploded"
Benjamin Loison
  • 3,782
  • 4
  • 16
  • 33
krecker
  • 21
  • 3