how to override timestamp field coming from json in logstash

Question

I have the following below kind of json present with me to be dumped in elastic search using filebeat {"@timestamp":"2017-02-10T06:30:51.424Z","beat":{"hostname":"myhostname","name":"mydevice-logger","version":"5.2.0"},"fields":{"device_type":"mydevice","env":"staging"},"metricset":{"module":"system","name":"cpu","rtt":211},"system":{"cpu":{"cores":4,"idle":{"pct":0.000000},"iowait":{"pct":0.000000},"irq":{"pct":0.000000},"nice":{"pct":0.000000},"softirq":{"pct":0.000000},"steal":{"pct":0.000000},"system":{"pct":0.000000},"user":{"pct":0.000000}}},"tags":["automata","box"],"type":"metricbeat-test-log"}

my logstash( version 5.1.1) config contains, input, filter and output like below -

input { 
  beats {
        port => 5046
        codec => json
  }
}

filter {
    if ...{}
    else if [type] == "metricbeat-test-log" {

      date {
        match => ["@timestamp", "ISO8601"]
      }

      }
    }

}


output {
    if ...{}
    else if [type] == "metricbeat-test-log" {
        stdout { codec => rubydebug   }
    }
}

The type is right however the date filter is not working . The @timestamp finally takes current timestamp always . I want to replace it with original @timestamp present in json.

Are you really using Filebeat to write Metricbeat's JSON data to Elasticsearch? — A J, Feb 13 '17 at 14:28
Yes the reason is its an offline device. I collect metricbeat log and putting them in to elastic search — Ankit Kulkarni, Feb 13 '17 at 14:32
though I can simply use a python script to dump the whole json into ES however I find using filebeat more practical for my case — Ankit Kulkarni, Feb 13 '17 at 14:33

score 0 · Answer 1 · answered Feb 14 '17 at 16:04

0

You should use the target setting in the date filter :

From https://www.elastic.co/guide/en/logstash/5.1/plugins-filters-date.html#plugins-filters-date-target

Target

Value type is string

Default value is "@timestamp"

Store the matching timestamp into the given target field. If not provided, default to updating the @timestamp field of the event.

answered Feb 14 '17 at 16:04

Bren

136
1
9

bren , I also thought same and had tried it , it gives `_dateparsefailure` because it can not find the @timestamp field . @timestamp in logs is not usually a string but instead an object . More details discussed in https://discuss.elastic.co/t/replace-timestamp-with-actual-timestamp-from-log-file/72017 – Ankit Kulkarni Feb 15 '17 at 06:42

score 0 · Accepted Answer · answered Feb 16 '17 at 06:26

I have got the answer following this old thread https://discuss.elastic.co/t/replace-timestamp-with-actual-timestamp-from-log-file/72017 which explains that the "@timestamp":"2017-02-10T06:30:51.424Z" in log line above is a JSON representation of the @timestamp field and its value. Following the suggestions I added the json configuration in filebeat and it worked for me .

- input_type: log
  paths:
    - /var/logs/mylogs/*.log
  fields:
    environment: testing
  document_type: test-metric-document

  json.keys_under_root: true   # added this line 
  json.overwrite_keys: true    # added this line

Though I am not happy with this solution as my real need was to get the both timestamp, the logstash event one(in some other variable ) and the timestamp from log in @timestamp.

how to override timestamp field coming from json in logstash

2 Answers2