logstash filter , unfiltered lines

Question

I am new to Logstash filter and going through different blogs and links to understand in detail. I have few questions which are still unanswered.

. If my log file has different log pattern e.g.

2017-01-30 14:30:58 INFO ThreadName:33 - {"t":1485786658088,"h":"abcd1234", "l":"INFO", "cN":"org.logstash.demo", "mN":"getNextvalue", "m":"fetching next value"} 2017-01-30 14:30:58 INFO AnotherThread:33 -my log pattern is different

I have below filter which is successfully filtering line 1 of the log

 grok
 {  
      match => [ "message", "%{TIMESTAMP_ISO8601:LogDate} %{LOGLEVEL:loglevel} %{WORD:threadName}:%{NUMBER:ThreadID} - %{GREEDYDATA:Line}" ] 
  }
  json
  {
      source => "Line" 
  }

1. what will happen with the lines which can not be filtered using filter pattern?
2. Is there any way to capture all the lines which were not filtered and send to elasticSearch ?
3. Is there any good reading material where I can read about Input, Filter, Output plugins with the examples ?

Answer 1

To answer your questions:

1. The lines which cannot be filtered using grok would end up in a grok_parsefailure. Make sure you handle it by dropping the lines which don't actually match the filter criteria.

2. As far as I know you can't capture them separately and push it to ES. Maybe for this, you can have multiple grok patterns so that you can filter it out and send it to different ES indices thereafter.

3. I've added the links in the comment above.

This SO could come in handy. Hope it helps!

Answer 2

As @darth_vader points out, you'll get a "grok_parsefailure" tag on each document that doesn't match your pattern(s) in a grok{} filter. However, how you handle this failure is up to you.

By default, all the events will fall through to your output{} section, which presumably would send them to elasticsearch. You could also have a conditional output{} section, which sent parsed logs to one output and unparsed logs to another (a file{} output, or a different index, or...).

As for examples, the official doc tends to include incomplete fragments (at best), so you're probably going to find better examples in random internet blogs.