Logstash grok match pattern for message field

Question

my log data is like,

log messages in the file .

There are total 4 lines are there(Starting from Date with Time).

My grok pattern is:

grok { 
match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg} " } }

Problem is:

I am getting only some data of msg(GREEDYDATA) filed.

EX:

Below data is missing when the 4th line parsing

missed data

log is :

2015-01-31 15:58:57,400 [9] ERROR NCR.AKPOS.Enterprise_Comm.EventSender - EventSender.SendInvoice() Generate Message Error: System.ArgumentNullException: Value cannot be null.
Parameter name: value
   at System.Xml.Linq.XAttribute..ctor(XName name, Object value)
   at NCR.AKPOS.Enterprise_Comm.MessageHandlerObjecttoXMLHelper.CreateXMLFromInvoice(Invoice invoice, Customer_ID customer_id

score 2 · Answer 1 · answered Jul 10 '15 at 12:08

2

Log stash typically parses each line at a time. For java exceptions you need to look at the multiline plugin. See an example here: https://gist.github.com/smougenot/3182192

Your grok format seems ok, but without an example cannot be tested. You can use the grok debugger app to test out your patterns. https://grokdebug.herokuapp.com/

answered Jul 10 '15 at 12:08

Yogesh_D

17,656
10
41
55

I already check with the GrokDebugger @Yogesh_D. I find the below pages for my problem but am unable to solve my problem http://stackoverflow.com/questions/22172879/how-to-process-logstash-multiline-filters-message-field – Anilkumar Bathula Jul 10 '15 at 12:19
Well you need to edit your question to have the log you are trying to parse, so that we can have a go at it – Yogesh_D Jul 10 '15 at 12:31
Please see the edited post add the log at end of the post – Anilkumar Bathula Jul 10 '15 at 12:41

score 2 · Answer 2 · edited May 04 '18 at 02:58

Just remove the trailing white spaces from %{GREEDYDATA:msg} " } to %{GREEDYDATA:msg}"}

So, total filter configuration is:

filter {
multiline{
        pattern => "^%{TIMESTAMP_ISO8601}"
        what => "previous"
        negate=> true
    }
# Delete trailing whitespaces
  mutate {
    strip => "message"
  }
# Delete \n from messages
mutate {
    gsub => ['message', "\n", " "]
}

# Delete \r from messages
mutate {
    gsub => ['message', "\r", " "]
}
grok { 
  match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg}" } 
}
if "Exception" in [msg] {
 mutate {
  add_field => { "msg_error" => "%{msg}" }
}
}
}

Logstash grok match pattern for message field

2 Answers2