Multiline log parsing without last char

Question

Hello I need parse a log. I have a problem when log has more than one line (in case of error) For example: If no errors

Last error code [0], message [No errors encountered.]

If conversion error

Last error code [8425], message [ERROR: Writer execution failed. 
Database error: [0] [
FnName: Execute -- [DataDirect][ODBC SQL Server Wire Protocol driver][Microsoft SQL Server]Conversion failed when converting the varchar value 'amica' to data type int., SQLSTATE [22005]
FnName: Execute -- [DataDirect][ODBC lib] Function sequence error, SQLSTATE [S1010]]]

I need to get log between [ ]

Base on first example it should be:

No errors encountered.

base on second example it should be:

ERROR: Writer execution failed. Database error: [0] [FnName: Execute -- [DataDirect][ODBC SQL Server Wire Protocol driver][Microsoft SQL Server]Conversion failed when converting the varchar value 'amica' to data type int., SQLSTATE [22005]FnName: Execute -- [DataDirect][ODBC lib] Function sequence error, SQLSTATE [S1010]]

I'm writing my script in awk. Is it possible to do this using regular expressions? I'm not familiar with regular expressions, but I tried something like this

(?<=, message \[).+\n

but it doesn't work with log which has many lines, and I don't know how to cut last one char "]"

*Is it possible to do this using regular expressions?* I would discourage regular expression usage here as for me it looks like you are working with nested brackets and this is generally tricky, see https://stackoverflow.com/questions/14952113/how-can-i-match-nested-brackets-using-regex — Daweo, Sep 09 '21 at 19:05

score 1 · Answer 1 · answered Sep 09 '21 at 21:01

It's not possible with a BRE or ERE as used by the mandatory POSIX tools (e.g. sed and awk) but might be possible with a PCRE using some other tool (e.g. perl), I really have no idea about that as I've never used PCREs.

You could always brute-force it by counting [ and ]s though, e.g. using any awk in any shell on every Unix box:

$ cat tst.awk
{ rec = (NR>1 ? rec RS : "") $0 }
END {
    numChars = length(rec)
    for (charNr=1; charNr<=numChars; charNr++) {
        char = substr(rec,charNr,1)
        if ( char == "[" ) {
            if ( ++depth == 1 ) {
                str = ""
            }
        }
        str = str char
        if ( char == "]" ) {
            if ( --depth == 0 ) {
                if ( ++numStrs % 2 == 0 ) {
                    gsub(/^\[|]$|\n/,"",str)
                    print str
                }
            }
        }
    }
}

$ awk -f tst.awk file
No errors encountered.
ERROR: Writer execution failed. Database error: [0] [FnName: Execute -- [DataDirect][ODBC SQL Server Wire Protocol driver][Microsoft SQL Server]Conversion failed when converting the varchar value 'amica' to data type int., SQLSTATE [22005]FnName: Execute -- [DataDirect][ODBC lib] Function sequence error, SQLSTATE [S1010]]

The above does have to read your whole input file into memory at once. You could alternatively keep a rolling buffer that you read input lines into and reset after every print of str - left as an exercise!

The input file used to test the above is:

$ cat file
Last error code [0], message [No errors encountered.]
Last error code [8425], message [ERROR: Writer execution failed.
Database error: [0] [
FnName: Execute -- [DataDirect][ODBC SQL Server Wire Protocol driver][Microsoft SQL Server]Conversion failed when converting the varchar value 'amica' to data type int., SQLSTATE [22005]
FnName: Execute -- [DataDirect][ODBC lib] Function sequence error, SQLSTATE [S1010]]]

score 0 · Answer 2 · answered Sep 09 '21 at 21:36

Here, an awk script:

{
    # Count open and close brackets
    line = $0
    nb_open = gsub(/\[/, "x", line)
    line = $0
    nb_close = gsub(/\]/, "x", line)
}
inError == 0 && nb_open == nb_close {
    # not in a block and brackets are balanced
    errorLine = ""
    print "No errors encountered."
    next
}
inError == 0 && nb_open != nb_close {
    # not in a block and brackets are unbalanced
    inError = 1
    errorLine = $0
    # remove error header
    gsub(/^.*, message \[ERROR: /, "ERROR:", errorLine)
    # initialize total counters
    nb_open_tot = nb_open
    nb_close_tot = nb_close
    next
}
{
    # in a block, add brackets to the total counters
    nb_open_tot  += nb_open
    nb_close_tot += nb_close
    # add the current line to the future error line
    errorLine = errorLine " " $0
}
inError == 1 && nb_open_tot != nb_close_tot {
    # continue (current line already added)
    next
}
inError == 1 && nb_open_tot == nb_close_tot {
    # remove close bracket
    gsub(/\]$/, "", errorLine)
    # print error line
    print errorLine
    # reset variables
    errorLine    = ""
    nb_open_tot  = 0
    nb_close_tot = 0
    inError      = 0
    next
}
END {
    # End of block not found ?
    if (errorLine) {
        print errorLine
    }
}

Multiline log parsing without last char

2 Answers2