grok parse multiple lines, for example exception stack trace

Question

I have multiple(three) types of log in my log file. One of the types has some own prints + exception stack trace. The example is listed below:

Multiple lines example:
    2018-04-27 10:53:17 [http-nio-8088-exec-4] - ERROR - app-info-exception-info - params:{"cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"} java.lang.NullPointerException: null
        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]
        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

I have a patterns to parse it, it is:

#pattern:
(?<timestamp>[\d\-\s\:]+)\s\[(?<threadname>[\w\-\d]+)\]\s-\s(?<loglevel>[\w]+)\s\-\s(?<appinfo>app-info-exception-info)\s-\s(?<params>params):(?<jsonstr>[\"\w\d\,\:\.\{\}]+)\s(?<exceptionname>[\w\d\.]+Exception):\s(?<exceptiondetail>[\w\d\.]+)\n\t(?<extralines>at[\s\w\.\d\~\?\n\t\(\)\_\[\]\/\:\-]+)\n

Pattern has error(actually not error, but not parse wholely or as expected) in parse the multiline exception stack trace, mostly in last two parts(exceptiondetail (null in this case) and extralines(those lines starting with space or tabs plus 'at', or lines after first line of stack trace)). Any better idea than I did?

In filebeat.yml, I have following configured:

# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: '^[[:space:]]'

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: false

  multiline.match: after

Any idea to improve parse multiple lines (exception stacktrace)?

score 9 · Accepted Answer · answered May 24 '18 at 23:25

How about making it simpler? assigning extra data (all lines start with at) to GREEDYDATA into a single field using (?m)?

For example, if this is your log,

2018-04-27 10:53:17 [http-nio-8088-exec-4] - ERROR - app-info-exception-info - params:{"cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"} java.lang.NullPointerException: null
        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]
        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

You can parse it as,

%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:threadname}\] - %{LOGLEVEL:loglevel} - app-info-exception-info - params:%{SPACE}\{\"%{DATA:jsondata}\"\} %{DATA:excentionname}: %{DATA:exceptiondetail}\n(?m)%{GREEDYDATA:extralines}

which will output,

{
  "timestamp": [
    [
      "2018-04-27 10:53:17"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "MONTHNUM": [
    [
      "04"
    ]
  ],
  "MONTHDAY": [
    [
      "27"
    ]
  ],
  "HOUR": [
    [
      "10",
      null
    ]
  ],
  "MINUTE": [
    [
      "53",
      null
    ]
  ],
  "SECOND": [
    [
      "17"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "threadname": [
    [
      "http-nio-8088-exec-4"
    ]
  ],
  "loglevel": [
    [
      "ERROR"
    ]
  ],
  "SPACE": [
    [
      ""
    ]
  ],
  "jsondata": [
    [
      "cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"
    ]
  ],
  "excentionname": [
    [
      "java.lang.NullPointerException"
    ]
  ],
  "exceptiondetail": [
    [
      "null"
    ]
  ],
  "extralines": [
    [
      "        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]\n        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]\n        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]"
    ]
  ]
}

you can replace (?m) with %{SPACE} to break each line that start with at into its own field as well.

It is a good suggestion. What does (?m) mean? Does that mean 'multiple lines begins to be parsed?'? — user84592, May 25 '18 at 01:08
that is kind of correct. `m` is multiline modifier, please have a look https://stackoverflow.com/questions/918806/difference-between-regular-expression-modifiers-m-and-s — Sufiyan Ghori, May 25 '18 at 01:13

score 0 · Answer 2 · answered May 24 '18 at 06:33

0

I think you could do it that way and tell that every new line will begin with a timestamp:

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after

answered May 24 '18 at 06:33

Evaldas Buinauskas

13,739
11
55
107

grok parse multiple lines, for example exception stack trace

2 Answers2