Custom Grok Pattern for logs

Question

So here is a sample of my log:

23:28:32.226 WARN  [MsgParser:ListProc-Q0:I5]   Parsing error
Error mapping the fieldAdditional Information: 

    at com.authentic.mapper.parsing.LengthVar.readBytes(LengthVar.java:178)
    at com.authentic.mapper.parsing.GrpLengthVar.read(GrpLengthVar.java:96)
    at com.authentic.mapper.parsing.GrpLengthVar.read(GrpLengthVar.java:119)
    at com.authentic.mapper.parsing.MsgParser.processReadEnumeration(MsgParser.java:339)
    at com.authentic.mapper.parsing.MsgParser.parseIncomingMessageBody(MsgParser.java:295)
    at com.authentic.mapper.MapperMgr.parseMsg(MapperMgr.java:1033)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.parseMessage(AbstractConnectionHandler.java:4408)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.plainMessageReceivedEvent(AbstractConnectionHandler.java:2031)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.messageReceivedEvent(AbstractConnectionHandler.java:1911)
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:801)
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:282)
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:261)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.processEventQueue(AbstractConnectionHandler.java:4110)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.access$100(AbstractConnectionHandler.java:320)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler$ConnectionHandlerRunner.execute(AbstractConnectionHandler.java:416)
    at com.authentic.architecture.actions.ListProcessor.suspend(ListProcessor.java:1130)
    at com.authentic.architecture.actions.ListProcessor.run(ListProcessor.java:775)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: "^123"
    at java.lang.NumberFormatException.forInputString(Unknown Source)
    at java.lang.Integer.parseInt(Unknown Source)
    at java.lang.Integer.parseInt(Unknown Source)
    at com.authentic.mapper.parsing.LengthVar.readBytes(LengthVar.java:170)
    ... 17 more

I have to parse this logs into following fields: timestamp, log-level, logger, msg, stacktrace.

i have used the multiline filter:

multiline {
pattern => "%{TIME:timestamp}"
negate => true
what => “previous”
}

and the pattern i used in grok filter:

match=>{"message"=>"%{TIME:timestamp} %{LOGLEVEL:loglevel} \s*\[%{DATA:logger}\]\s*%{GREEDYDATA:msg}\n*(?<stacktrace>(.|\r|\n)*)"}

i have checked it with http://grokconstructor.appspot.com/do/match. but got this matching error for stacktrace field.

please do suggest some suggestions. thanks in advance.

score 1 · Answer 1 · answered Aug 29 '17 at 13:50

1

You will need a multiline filter if you want to match the whole stacktrace. This multiline filter should work for you:

codec => multiline {
        pattern => "^%{TIME} "
        negate => true
        what => previous
    }

Explanation: Every line not starting with a timestamp (like 23:28:32.226) will be regocnized as part of the previous line. See also the docs on dealing with multilines.

Now to your pattern. Following works for me:

%{TIME:timestamp} %{LOGLEVEL:loglevel}  \[%{DATA:logger}\]   %{GREEDYDATA:message}\n(?<stacktrace>(.|\r|\n)*)

Pretty self explaining, I hope: Escaping braces like [ and ] with \[ and \], \n to match the newline. Also note the spaces between the entries.

For the last part (stacktrace) also see this question on how to match everything including newlines.

A full configuration could look something like this:

input {
  file {
    path => "/var/log/yourlog.log"
    start_position => "beginning"
    codec => multiline {
        pattern => "^%{TIME} "
        negate => true
        what => previous
    }
  }
}
filter {
  grok {
    match => [ "message", "%{TIME:timestamp} %{LOGLEVEL:loglevel}  \[%{DATA:logger}\]   %{GREEDYDATA:message}\n(?<stacktrace>(.|\r|\n)*)" ]
  }
}

Results on http://grokconstructor.appspot.com:

answered Aug 29 '17 at 13:50

Phonolog

6,321
3
36
64

hi @Phonolog please review my answer – aditya soni Aug 30 '17 at 09:58
Please [edit](https://meta.stackexchange.com/questions/21788/how-does-editing-work) your original question instead of posting another answer. – Phonolog Aug 31 '17 at 07:23
hi @Phonolog i have edited the same, please provide some solution now. – aditya soni Aug 31 '17 at 08:15
I think you're not using the mutliline field on http://grokconstructor.appspot.com right. You have to check the "negate the multiline regex" checkbox. This corresponds to `negate => true` in the config... – Phonolog Aug 31 '17 at 08:59
okay , so now this online tool showing the matched result for "stacktrace" also, BUT this parsing is not happening when i visualize it in KIBANA, in KIBANA it is not showing the name of field i.e stacktrace in the available fields tab...............and one more thing....in the online tool GROK contructor, if i put two lines of logs 1) first with exception stacktrace 2) second with normal log.. then it doesn't identifies the second line as multiline, it just add it into first stacktrace row only. – aditya soni Aug 31 '17 at 09:59
Well I have no machine to test here, I have to rely on the results of the online tool... Make sure you uncheck "Hide missing fields" in Kibana, otherwise I can't help you any further – Phonolog Aug 31 '17 at 17:54

Custom Grok Pattern for logs

1 Answers1