Using JSON with LogStash

Question

I'm going out of my mind here. I have an app that writes logs to a file. Each log entry is a JSON object. An example of my .json file looks like the following:

{"Property 1":"value A","Property 2":"value B"}
{"Property 1":"value x","Property 2":"value y"}

I'm trying desperately to get the log entries into LogStash. In an attempt to do this, I've created the following LogStash configuration file:

input {
  file {
    type => "json"
    path => "/logs/mylogs.log"
    codec => "json"
  }
}
output {
  file {
    path => "/logs/out.log"
  }
}

Right now, I'm manually adding records to mylogs.log to try and get it working. However, they appear oddly in the stdout. When I look open out.log, I see something like the following:

{"message":"\"Property 1\":\"value A\", \"Property 2\":\"value B\"}","@version":"1","@timestamp":"2014-04-08T15:33:07.519Z","type":"json","host":"ip-[myAddress]","path":"/logs/mylogs.log"}

Because of this, if I send the message to ElasticSearch, I don't get the fields. Instead I get a jumbled mess. I need my properties to still be properties. I do not want them crammed into the message portion or the output. I have a hunch this has something to do with Codecs. Yet, I'm not sure. I'm not sure if I should change the codec on the logstash input configuration. Or, if I should change the input on the output configuration.

score 49 · Answer 1 · edited Nov 19 '21 at 12:02

49

Try removing the json codec and adding a json filter:

input {
  file {
    type => "json"
    path => "/logs/mylogs.log"
  }
}
filter{
    json{
        source => "message"
    }
}
output {
  file {
    path => "/logs/out.log"
  }
}

You do not need the json codec because you do not want decode the source JSON but you want filter the input to get the JSON data in the @message field only.

edited Nov 19 '21 at 12:02

tshepang

12,111
21
91
136

answered Apr 08 '14 at 16:03

vzamanillo

9,905
1
36
56

1

Sweet! That worked. Except, it didn't help me get past my real JSON :(. I'll give you credit for the answer. I opened a secondary question at http://stackoverflow.com/questions/22944168/json-variants-log4j-with-logstash – user70192 Apr 08 '14 at 17:29
1

Hi ! I used this logstash configuration file `input { tcp { port => '9563' } } filter{ json{ source => "message" } } output { elasticsearch { hosts => [ "localhost:9200" ] } }` but still in elastic search my json is put as a string in message field as _`message:"{ \"container\": \"DOCKER\", \"msg\" : \"From python 26 oct\" }"`_ . I am unable get _`container`_ and _`msg`_ as different field in kibana. I have this extra field in kibana saying _tags:_jsonparsefailure_ . My json is _{ "container": "DOCKER", "msg" : "From python 26 oct" }_ – Tinkaal Gogoi Oct 26 '16 at 06:35
The logstash log `{:timestamp=>"2016-10-26T03:12:51.814000-0400", :message=>"Parsed JSON object/hash requires a target configuration option", :source=>"message", :raw=>"\"{ \\\"container\\\": \\\"DOCKER\\\", \\\"msg\\\" : \\\"From python 26 oct\\\" }\"", :level=>:warn}` – Tinkaal Gogoi Oct 26 '16 at 07:02
@vzamanillo the json filter is still going to be doing an unwanted/unnecessary parse of the JSON though, isn't it? I also can't get the default target to do what it's supposed to i.e. replace the event root (v 5.6.1), but that's probably a separate issue. – mrec Sep 29 '17 at 18:00

score 14 · Answer 2 · answered May 29 '17 at 07:06

By default tcp put everything to message field if json codec not specified.

An workaround to _jsonparsefailure of the message field after we specify the json codec also can be rectified by doing the following:

input {
  tcp {
    port => '9563'
  }
}
filter{
  json{
    source => "message"
    target => "myroot"
  }
  json{
    source => "myroot"
  }

}
output {
    elasticsearch {
      hosts => [ "localhost:9200" ]
    }
}

It will parse message field to proper json string to field myroot and then myroot is parsed to yield the json.

We can remove the redundant field like message as

filter {
  json {
    source => "message"
    remove_field => ["message"]
  }
}

score 3 · Answer 3 · answered Jul 17 '20 at 04:15

3

Try with this one:

filter {
  json {
        source => "message"
        target => "jsoncontent" # with multiple layers structure
  }
}

answered Jul 17 '20 at 04:15

Newt

787
8
15

Using JSON with LogStash

3 Answers3

Linked