In Logstash, how do I extract fields from a log event using the json filter?

Question

Logstash v2.4.1.

I'm sending JSON formatted logs to my Logstash server via UDP packet. The logs look something similar to this.

{
  "key1":"value1",
  "key2":"value2",
  "msg":"2017-03-02 INFO [com.company.app] Hello world"
}

This is my output filter

output {
  stdout {
    codec => rubydebug
  }
  file {
    path => "/var/log/trm/debug.log"
    codec => line { format => "%{msg}" }
  }
}

The rubydebug output codec shows the log like this

{
  "message" => {\"key1\":\"value1\", "key2\":\"value2\", \"msg\":\"2017-03-02 INFO [com.company.app] Hello world\"
}

and the file output filter also shows the JSON log correctly, like this

{"key1":"value1", "key2":"value2", "msg":"2017-03-02 INFO [com.company.app] Hello world"}

When I use the JSON code in the input filter, I get _jsonparsefailures from Logstash on "some" logs, even though different online JSON parsers parse the JSON correctly, meaning my logs are in a valid JSON format.

input {
  udp => {
    port => 5555
    codec => json
  }
}

Therefore, I'm trying to use the json filter instead, like this

filter {
  json => {
    source => "message"
  }
}

Using the json filter, how can I extract the "key1", "key2", and the "msg" fields in the "message?"

I tried this to no avail, that is, I don't see the "key1" field in my rubydebug output.

filter {
  json => {
    source => "message"
    add_field => {
      "key1" => "%{[message][key1]}"
    }
  }
}

Can you put the exact messages you receive on error? What version of Logstash are you using? Can you also provide the original message received by the server before being processed by Logstash? — Adonis, Mar 07 '17 at 17:52
The original message coming into the server is the JSON message I showed at the start of the post. "Most" messages go through fine, but messages with stack dumps result in the _jsonparsefailure when using the json input codec. — Chris F, Mar 07 '17 at 18:03
Can I have the exact Logstash output of the error message? And I meant a json which fails in logstash, as seen by the server, because obviously if the agent works normally on your well formed json, then something happens with the failed ones. — Adonis, Mar 07 '17 at 20:10
This is similar to [How to parse json in logstash /grok from a text file line?](http://stackoverflow.com/questions/33937936/how-to-parse-json-in-logstash-grok-from-a-text-file-line), which has the answer. — Chris F, Mar 07 '17 at 21:35
This only happens on HUGE log events, e.g., stack traces. Logstash truncates the JSON event, and hence complains about _jsonparsefailure. I even set the buffer_size, and switch ti TCPv6 to no avail. — Chris F, Mar 10 '17 at 14:32
Which is why I suggested to use the multiline codec to be sure to aggregate several events into a single one: https://www.elastic.co/guide/en/logstash/5.0/plugins-codecs-multiline.html — Adonis, Mar 10 '17 at 15:40

Adonis · Answer 1 · 2017-03-10T16:26:41.233

I would suggest you to start with one of the two configuration below (I use the multiline codec to concatenate the input into a json, because otherwise logstash will read line by line, and one line of a json is not a valid json), then either filter the json, or use the json codec, and then output it to wherever it is needed. You will still have some configuration to do, but I believe it might help you to get started:

input{
  file {
     path => "/an/absolute/path/tt2.json" #It really has to be absolute!
     start_position => beginning
     sincedb_path => "/another/absolute/path" #Not mandatory, just for ease of testing

   codec =>   multiline{
    pattern => "\n"
        what => "next"

     }
  }
}

filter{
   json {
     source => "multiline"
   }
}
output {
  file {
    path => "data/log/trm/debug.log"
  }

 stdout{codec => json}
}

Second possibility:

input{
  file {
    path => "/an/absolute/path/tt2.json" #It really has to be absolute!
         start_position => beginning
         sincedb_path => "/another/absolute/path" #Not mandatory, just for ease of testing

   codec =>   multiline{
    pattern => "\n"
        what => "next"

     }
   codec => json{}
  }
}


output {
  file {
    path => "data/log/trm/debug.log"
  }

 stdout{codec => json}
}

Edit With the udp input I guess it should be (not tested):

input {
  udp => {
    port => 5555

   codec =>   multiline{ #not tested this part
    pattern => "^}"
        what => "previous"

     }
   codec => json{}
  }
}

Thanks, but my logs come in via UDP, so the file input filter does not make sense for me. I've updated the original post to show this. — Chris F, Mar 07 '17 at 17:41
See my edit, I guess you could use something that looks more or less like that, note that I did not test as I don't have any such environment for testing — Adonis, Mar 10 '17 at 16:27

In Logstash, how do I extract fields from a log event using the json filter?

1 Answers1