14

I'm having issues with grok parsing. In ElasticSearch/Kibana the lines I match come up with the tag _grokparsefailure.

Here is my logstash config :

input { 
    file { 
     type => logfile 
     path => ["/var/log/mylog.log"] 
    } 
  } 
filter { 
    if [type] == "logfile" 
    { 
      mutate {
      gsub => ["message","\"","'"]
      }  

    grok 
        { match => { "message" => "L %{DATE} - %{TIME}: " } } 
    } 
} 

output { 
   elasticsearch { host => localhost port => 9300 } 
}

lines/patterns I'm trying to match : L 08/02/2014 - 22:55:49: Log file closed : " finished "

I tried the debugger on http://grokdebug.herokuapp.com/ and it works fine, my pattern matches correctly.

Lines I want to parse might contain double quotes, and I've read there can be issues with the way grok handles and escapes them. So I tried to mutate to replace " with ' to avoid issues but no luck.

Any ideas ? How can I debug this ?

Thanks

lepolac
  • 263
  • 1
  • 3
  • 9
  • 13
    Is there something else in your mylog.log? Because any line that doesn't match will generate a `_grokparsefailure`. A good strategy for debugging this is to create a test file that has the expected log in it, and use a config file with `input { stdin{} }` and `output {stdout { codec => rubydebug } }` and then do logstash -f test_conf < test_file and see what's going on. If you do that and post the input/output, it might be easier to help. As is, your filter is correct for the line given and outputs correctly, although you aren't capturing the `grok` results... for example`{%DATE:date}` – Alcanzar Aug 18 '14 at 18:56
  • Hi, There are other things in the log on which I would expect grokparsefailure yes. Tried using ./logstash -e 'input { stdin{} } output {stdout { codec => rubydebug } }' < mylog.log and got outputs things like : { "message" => "L 08/02/2014 - 22:55:49: Log file closed", "@version" => "1", "@timestamp" => "2014-08-18T19:14:38.033Z", "host" => "lepostack" } Which, seems fine to me :( – lepolac Aug 18 '14 at 19:17
  • @Alcanzar This comment is worth an answer for it answers the question "How can I debug this". I'd give you an upvote, because it helped me. – Matthias M Nov 10 '15 at 14:54

1 Answers1

11

Found out the issue, it was around double quotes.

Needed to use simple quote to define the grok filter, and escape double quotes.

match => { 'message' => 'L %{DATE:date} - %{TIME:time}: \"string_between_doublequotes\" '
lepolac
  • 263
  • 1
  • 3
  • 9