How to handle non-matching Logstash grok filters

Question

I am wondering what the best approach to take with my Logstash Grok filters. I have some filters that are for specific log entries, and won't apply to all entries. The ones that don't apply always generate _grokparsefailure tags. For example, I have one grok filter that's for every log entry and it works fine. Then I have another filter that's for error messages with tracebacks. The traceback filter throws a grokparsefailure for every single log entry that doesn't have a traceback.

I'd prefer to have it just pass the rule if there isn't a match instead of adding the parsefailure tag. I use the parsefailure tag to find things that aren't parsing properly, not things that simply didn't match a particular filter. Maybe it's just the nomenclature "parse failure" that gets me. To me that means there's something wrong with the filter (e.g. badly formatted), not that it didn't match.

So the question is, how should I handle this?

Make the filter pattern optional using ?
(ab)use the tag_on_failure option by setting it to nothing []
make the filter conditional using something like "if traceback in message"
something else I'm not considering?

Thanks in advance.

EDIT

I took the path of adding a conditional around the filter:

    if [message] =~ /took\s\d+/ {
        grok {
            patterns_dir => "/etc/logstash/patterns"
            match => ["message", "took\s+(?<servicetime>[\d\.]+)"]
            add_tag => [ "stats", "servicetime" ]
        }
    }

Still interested in feedback though. What is considered "best practice" here?

score 38 · Answer 1 · edited Mar 14 '19 at 10:34

38

When possible, I'd go with a conditional wrapper just like the one you're using. Feel free to post that as an answer!

If your application produces only a few different line formats, you can use multiple match patterns with the grok filter. By default, the filter will process up to the first successful match:

grok {
    patterns_dir => "./patterns"
    match => {
        "message" => [ 
              "%{BASE_PATTERN} %{EXTRA_PATTERN}",
              "%{BASE_PATTERN}",
              "%{SOME_OTHER_PATTERN}"
        ]
    }
}

If your logic is less straightforward (maybe you need to check the same condition more than once), the grep filter can be useful to add a tag. Something like this:

grep {
    drop => false #grep normally drops non-matching events
    match => ["message", "/took\s\d+/"]
    add_tag => "has_traceback"
}


...

if "has_traceback" in [tags] {
    ...
}

edited Mar 14 '19 at 10:34

IMSoP

89,526
13
117
169

answered Jan 02 '14 at 20:31

rutter

11,242
1
30
46

2

This is in violation of the DRY principle. It can't be that you first have to match a message to see if you should match a message. Especially since it's not a heuristic; the message has to match if the condition is met, otherwise the grok fails. – malthe Mar 14 '14 at 15:44
9

I see your point, but also notice you offer no alternative solution. With a few more months of experience, I know that an empty `tag_on_failure` may work for some use cases... but I also know the behavior on parse failures is barely documented and subject to change in the future. – rutter Mar 17 '14 at 19:47
match => [ "message", "a", "message", "b", "message", "c" ] results in an error. – Stefan K. Oct 06 '15 at 09:51
4

oh, it's match => { "message" => [ "Duration: %{NUMBER:duration}", "Speed: %{NUMBER:speed}" ] } – Stefan K. Oct 06 '15 at 11:07
just an update on your link to the logstash docs about conditionals: https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals – tonyg Oct 23 '15 at 14:12
2

Thanks to @StefanK. for having found the correct syntax. – jmcollin92 Feb 09 '16 at 10:38

Gary Rogers · Answer 2 · 2014-08-25T20:44:22.297

24

You can also add tag_on_failure => [] to your grok stanza like so:

grok {
    match => ["context", "\"tags\":\[%{DATA:apptags}\]"]
    tag_on_failure => [ ]
}

grok will still fail, but will do so without adding to the tags array.

edited Aug 25 '14 at 20:44

answered Aug 25 '14 at 18:01

Gary Rogers

395
2
7

I think this is the most standard way to do it. – alfredocambera Apr 08 '16 at 15:23
Great this way you can create fail tags that are understandable also by people who don't know anything about grok – herm Oct 04 '17 at 12:41

score 9 · Answer 3 · answered Apr 18 '17 at 22:23

9

This is the most efficient way of doing this. Ignore the filter

filter {

        grok {
            match => [ "message", "something"]
    }

    if "_grokparsefailure" in [tags] {
            drop { }
        }
}

answered Apr 18 '17 at 22:23

JAR

393
4
8

score 4 · Answer 4 · answered Apr 23 '14 at 23:05

4

You can also do this

remove_tag => [ "_grokparsefailure" ]

whenever you have a match.

answered Apr 23 '14 at 23:05

Tony Wong

49
1

How to handle non-matching Logstash grok filters

4 Answers4

Linked