Logstash configuration

Question

I am trying to parse, and analyse my logs with Logstash, and output them to elasticsearch. Logstash treats each line in the log file as a new log entry, but the problem is that one log entry could be in multiple lines. I want configure the filter to extract the useful data (date,thread,query...).

Answer 1

You'll need to look at the multiline filter - http://logstash.net/docs/1.4.1/filters/multiline

I've done something similar with MariaDB logs which can be multiline - the regular expressions will be different but have a look at http://simonhanmer.co.uk/processing-mysql-mariadb-galera-logs-for-logstash/ for some ideas

OK, extracting the thread is fairly straightforward since it's delimited by [], but the query is a little more difficult. If it's always preceded by the line shown you could use something like this

input { 
    pipe {
      command => 'type C:\Users\MEGDICHE\Downloads\default.log'
     }
}

filter{
    multiline {
        pattern => "^%{MONTH} %{MONTHDAY}"
    negate  => true
    what    => "previous"
    }

    grok {
        match => [ 'message', "(?m)^%{MONTH} %{MONTHDAY}.*\[%{GREEDYDATA:thread}\]:%{GREEDYDATA:rest_of_message}" ]
    }

    if [rest_of_message] =~ /select/ {
        grok {
            match => [ 'message', '(?m).*\nThe query before.*\n%{GREEDYDATA:query}$' ]
        }
    }

    mutate {
        remove_field => [ 'rest_of_message']
    }
}

output { 
  elasticsearch { host => localhost }
  stdout { codec => rubydebug } 
 }