3

I've been going at this for weeks now and I can't seem to wrap my head around what's wrong about this.

I'm trying to get all of these lines to fit into a multiline match with grok, but it only picks up the last one, and even discards the digit at the beginning of the line.

11:31:03.936    <           :     1>    5: Load times per type (ms):
12: aaaaaa.aaaaaaaaa.aaaaaaa.aaaaaaa
1: bbbb.bbbb.bbbbbbbbbbbbb.bbbbbbbbb
3: cccc.cccccccc.ccccccccccc.cccccc
64: ddd.dddddddddddd.ddddddd.ddddddd

Expected result:

message_processed = Load times per type (ms):
12: aaaaaa.aaaaaaaaa.aaaaaaa.aaaaaaa
1: bbbb.bbbb.bbbbbbbbbbbbb.bbbbbbbbb
3: cccc.cccccccc.ccccccccccc.cccccc
64: ddd.dddddddddddd.ddddddd.ddddddd

Actual result:

message_processed = ddd.dddddddddddd.ddddddd.ddddddd

I'm using the following grok pattern:

grok {
        match => [ "message" , "%{TIME:time}.*%{NUMBER:loglevel}:\s%{GREEDYDATA:message_processed}" ]
    }

It is being shipped to logstash with filebeat on a windows server with the following multi-line config in filebeat.yml:

multiline.pattern: ^[0-9]{2}\:[0-9]{2}\:[0-9]{2}
multiline.negate: true
multiline.match: after

I've tried using (?m) flag but to no avail, and using multi-line codec with filebeat is a no-go according to the official documentation.

What am I doing wrong?

Atombob
  • 114
  • 1
  • 9

1 Answers1

2

Try the following:

%{TIME:time}.*%{NUMBER:loglevel}:\s(?<message_processed>(.|\‌​r|\n)*)

As pointed out here GREEDYDATA won't match newlines while (?<message>(.|\r|\n)*) will. So the above works for me.

See the results for your example with your multiline config on https://grokconstructor.appspot.com/do/match: results

Phonolog
  • 6,321
  • 3
  • 36
  • 64
  • I tried that a while back, and it didn't seem to make any difference. But for the sake of being thorough i changed it again so that now it reads: `"%{TIME:time}.*%{NUMBER:loglevel}:\s(?(.|\‌​r|\n)*)"` And now, it fails to load the logs at all. If i change it back to the grok expression, logs start loading again. – Atombob Jan 10 '18 at 08:46