1

I am trying to measure the latency involved with using the ELK stack. I am logging in my test application and I want to find out how long it takes before it appears in ElasticSearch. I understand this will only be a rough estimate and is very specific to my environment.

How can I measure the latency between my app/logstash/elasticsearch?

EDIT: I am following suggestion and enabled _timestamp but I don't see the field in my records.

{
 logaggr : {
    order : 0,
    template : "logaggr-*",
    settings : {},
    mappings : {
        logaggr : {
            date_detection : false,
            _timestamp : {
                enabled : true,
                store: true
            },
            properties : {
                level : {
                    type : "string"
                },
                details : {
                    type : "string"
                },
                logts : {
                    format : "yyyy-MM-dd HH:mm:ss,SSS",
                    type : "date"
                },
                classname : {
                    type : "string"
                },
                thread : {
                    type : "string"
                }                   
            }
        }
    },
    aliases : {}
 }
}

Thanks in advance!

Community
  • 1
  • 1
Raylite3
  • 837
  • 2
  • 11
  • 22

1 Answers1

1

There are three timestamps that will answer your question:

  1. the log file timestamp, e.g. when the application wrote the information. Make sure your server's clock is correct.
  2. @timestamp, which is set by logstash to the time when it receives the log.
  3. _timestamp, which elasticsearch can set to the time when it receives the log. This setting must be enabled in elasticsearch.

Between these three, you can track the progress of your logs through ELK.

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • This should help me answer my question except I can't seem to get the _timestamp populated. I enabled _timestamp in my mapping and also set "store": true for this property but I don't see it when I query by ID Is there an additional flag to access it as well? – Raylite3 Jan 14 '15 at 18:48
  • You may have to add it explicitly to your field list. – Alain Collins Jan 14 '15 at 20:38
  • I am not able to get _timestamp populated even though my template shows that _timestamp is enabled. I have edited my post to show the template. Any ideas? – Raylite3 Jan 31 '15 at 00:35
  • Your mapping creates the field and indexes it (so you can search on it), but only "stored" fields are returned in queries. One more mapping update! – Alain Collins Jan 31 '15 at 16:05
  • Still could not see _timestamp even after I added "store": true. Until I realized from this http://stackoverflow.com/questions/13744233/elasticsearch-timestamp post that the _timestamp must be at the same level as properties. Updated my post to show the template that works for me now. I can see _timestamp !! – Raylite3 Feb 02 '15 at 22:26