Sometimes logstash replay all logs

Question

To send my service logs to kafka, I use logstash(6.5.3). But server restart and when logstash start sometimes send all past logs to kafka. I don't know why did it.

(ex)

log 1 -> completed
log 2 -> completed
log 3 -> completed
log 4
log 5

server restart and sometimes logstash restart

log 1 -> replay
log 2 -> replay
log 3 -> replay
log 4 -> send
log 5 -> send

but I expected:

log 4 -> send
log 5 -> send

This is my logstash conf:

input {
   file {
       type => "log"
       path => [...]
       sincedb_write_interval => 1
       start_position => "beginning"
   }
   .
   .
}

score 0 · Answer 1 · answered Jan 10 '19 at 06:23

Try below config

input {
  file {
    path => "*.log"
    type => "log"
    start_position => "beginning"
    ignore_older => 0
    sincedb_path => "/usr/share/log/sincedb/log.sincedb"
    sincedb_write_interval => 1
  }
}

where sincedb_path needs to be a file where logstash has write permission for the registry. It writes the last read details in this file

sincedb_write_interval defines how often logstash should write the sincedb registry. A larger value puts you at risk in logstash were to crash.

Ref: Logstash File input: sincedb_path

https://discuss.elastic.co/t/how-does-sincedb-work-for-a-logstash-reading-multiple-files-from-a-single-directory/99456/4

Sometimes logstash replay all logs

1 Answers1