Need to parse a log file against a Blacklist

Question

Afternoon Experts.

I have a Log file that is populated with Errors. Here is a Condensed Version :

*****2015-05-19 17:10:23,536 [ajp-0.0.0.0-8009-31] (TqlBuilder.java:347) ERROR - DUMMY From TqlBuilder getQueryDefinition
java.lang.Exception
xxxxxxx
xxxxxxxxx
xxxxxxx
2015-05-19 17:10:23,588 [ajp-0.0.0.0-8009-31] (TqlBuilder.java:348) ERROR - DUMMY From SqlBuilder 
java.lang.Exception
xxxx
xxxx
xxxx
xxxxxxxx

2015-05-19 17:10:23,598 [ajp-0.0.0.0-8009-31] (TqlBuilder.java:351) ERROR - Bla Bla Bla java.lang.Exception xxxx xxxx xxxx xxxxxxxx**

I would like to come up with a Bash-Based Solution (been trying with grep, but with limited success) to pass this Log through a Blacklist File populated with (list of examples errors below), to clean the original file of these errors and produce a new file with all the errors from the blacklist cleared, along with the Lines that follow.

Here is an Example of my Blacklist

(TqlBuilder.java:347) ERROR - DUMMY From TqlBuilder getQueryDefinition
(TqlBuilder.java:348) ERROR - DUMMY From SqlBuilder

Using plain grep I was able to remove just the lines, but I need a Solution to remove the lines that follow the Error. I thought of having grep remove all lines between the Date 'Tags' but I was not able to figure out how to go about that.

My Expected output for the new, cleaned file after applying the original against the blacklist, would be:

**2015-05-19 17:10:23,598 [ajp-0.0.0.0-8009-31] (TqlBuilder.java:351) ERROR - Bla Bla Bla
    java.lang.Exception
    xxxx
    xxxx
    xxxx
    xxxxxxxx**

[This post](http://stackoverflow.com/questions/3717772/regex-grep-for-multi-line-search-needed) may be helpful. Have you seen it? — Alp, Jun 18 '15 at 20:50

score 0 · Answer 1 · answered Jun 18 '15 at 14:41

Grep will not be able to do this by itself. You will need to either use a compiled program or script of some sort (language not important). While you could do this using multi-line matching regexes. It would be more straight forward to track state when walking through the file. In pseudo code something like:

inBlackList = false
while ( read next line )
{
    if ( ( inBlackList == false ) && ( current line starts blacklisted entry ) )
    {
        inBlackList = true
    }
    else if ( ( inBlackList == true ) && 
              ( current line starts new error ) && ( current line does not start blacklisted entry ) )
    {
        inBlackList = false
    }

    if ( inBlackList == false )
    {
        output current line to new file
    }
}  #  End while

Need to parse a log file against a Blacklist

1 Answers1