0

i looking for many many time and also consult in here to found one solution to resolve problem json in elastic using logstash. my config here

input {
  tcp {
    port => 9000
  }
}
filter{
    json{
        source => "message"
        target => "doc"
    }
}
output {
  elasticsearch {
   hosts => ["localhost:9200"] 
   index => logstash-%{+YYYY.MM.dd}
  }
}

but my elastic still have string message document not json. Document like that

{
  "_index": "logstash-2017.05.12",
  "_type": "logs",
  "_id": "AVv8C4O4qok70-ifTOnm",
  "_score": null,
  "_source": {
    "message": "{\"name\":\"abc\",\"id\":1494582167248}",
    "@version": "1",
    "@timestamp": "2017-05-12T09:42:47.263Z",
    "host": "172.0.0.1",
    "port": 53763
  },
  "fields": {
    "@timestamp": [
      1494582167263
    ]
  },
  "sort": [
    1494582167263
  ]
}

Any one can help me how to fix inorder to name and id filed is as an member propeter of _source . Im expect document log like

"_source": {
    "name":"abc",
    "id": 1494582167248
    "@version": "1",
    "@timestamp": "2017-05-12T09:42:47.263Z",
    "host": "192.168.2.251",
    "port": 53763
  }
Community
  • 1
  • 1
bapzangbo
  • 163
  • 1
  • 1
  • 5

1 Answers1

0

If you want to have those fields at the root of the parsed message (which will be at the root level of _source in ElasticSearch, you must remove the JSON target setting. That setting specifies a parent to the parsed fields that are extracted using the JSON filter:

{
  "@timestamp": "2017-05-12T11:58:40.897Z",
  "port": 61981,
  "@version": "1",
  "host": "10.0.2.2",
  "doc": {
    "name": "abc",
    "id": 1494582167248
  },
  "message": "{\"name\":\"abc\",\"id\":1494582167248}"
}

So remove the target setting, and make sure that your index setting is set in quotes:

input {
    tcp {
        port => 9000
    }
}

filter{
    json {
        source => "message"
    }
}

output {
    stdout { codec => "rubydebug" }
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
    }
}

Which results in:

{
  "@timestamp": "2017-05-12T11:56:51.187Z",
  "port": 61970,
  "@version": "1",
  "host": "10.0.2.2",
  "name": "abc",
  "id": 1494582167248,
  "message": "{\"name\":\"abc\",\"id\":1494582167248}"
}

To validate with ElasticSearch curl -XGET localhost:9200/logstash-2017.05.12/_search | jq . returns

{
  "_index": "logstash-2017.05.12",
  "_type": "logs",
  "_id": "AVv8hMTCjLo8wwWpi9R6",
  "_score": 1,
  "_source": {
    "@timestamp": "2017-05-12T11:56:51.187Z",
    "port": 61970,
    "@version": "1",
    "host": "10.0.2.2",
    "name": "abc",
    "id": 1494582167248,
    "message": "{\"name\":\"abc\",\"id\":1494582167248}"
  }
}
signus
  • 1,118
  • 14
  • 43