0

I am using LogStash which accepts data from a log file, which has different types of logs.

I tried this: 

filter {
  grok {
    match => { "message" => "%{WORD:tag} %{WORD:message} %{WORD:value}
  }
}

But it doesn't work.

Rohan Singh
  • 13
  • 4
  • In your grok filter, you are missing a " at the end. – baudsp Aug 19 '16 at 09:25
  • Does all the tags and message are always one word (only letters without any space or other type of characters)? – baudsp Aug 19 '16 at 09:27
  • @baudsp, I have tried to recreate a minimalistic example of my use case. Yes, it can be more characters as well. And I tried even with the double quote, but it doesn't work. I think since, my log file has multiple formats, going forward with grok won't be the right option. – Rohan Singh Aug 19 '16 at 09:31
  • Well grok is to parse a log line and create field in the json document created by logstash. If your use case is just to write the content of a log file in two other files, Logstash might not be the right option, since it adds a lot of complexity and overhead – baudsp Aug 19 '16 at 09:34
  • @baudsp, as I said, this is a minimalistic example of my use case where I removed a lot of complexity so as to keep it simple. My use case requires me to use Logstash. – Rohan Singh Aug 19 '16 at 09:35
  • You should have a look at this answer which shows how to validate JSON using a regexp: http://stackoverflow.com/questions/2583472/regex-to-validate-json – Val Aug 19 '16 at 10:02
  • @Val, I realised my mistake. And I was able to implement that as well. Thanks :). But, I just made another code, and it's I guess completely right, except a minor error which I am missing. Can you please have a look at it once? Here is the link: http://stackoverflow.com/questions/39064570/why-doesnt-this-print-values-in-logstash – Rohan Singh Aug 21 '16 at 17:57

1 Answers1

0

I am using the grok filter to check if the log line is of one format.

If the grok filter cannot parse the log line (such as with the json lines), _grokparsefailure will be added to the tags. You can then use this tag to differentiate between the two log type.

filter { 
    grok {
        match => {
        "message"=> 
            "tag: %{GREEDYDATA:tag} message: %{GREEDYDATA:message} value: %{WORD:value}"
        }
    }

    if "_grokparsefailure" in [tags] {
        json {
            source => message
        }
    }
}

To test your grok pattern, Grok Constructor is a good tool.

baudsp
  • 4,076
  • 1
  • 17
  • 35
  • is there any way I can do it with regex? Because, finally, I have to modify the contents of the logs too. Hence, a way with regex would be the best way out. Thanks! – Rohan Singh Aug 19 '16 at 09:57