How far along is logstash in processing a file

Question

I have logstash running with something like this:

input {
  file {
    path => ["/home/logdata/*.log"]
    codec => json {
        charset => "UTF-8"
        }
  }
}
output {
  elasticsearch {
        protocol => "http"
        host => "whatever"
        cluster => "recordings"
        index => "logstash-%{+YYYYMMdd}"
  }
}

Some of these files are many GB in size. How can I tell where logstash is in the process? Is there a progress indicator? Can I tell how many lines in or are left?

Amongst other things, I'm trying to figure out when to delete the original log files so knowing when logstash is finished would be very useful.

Answer 1

Logstash uses sincedb files to track the current read position of each input file. Understanding sincedb files from Logstash file input describes the format of those files, and the name of each sincedb file is devised from the filename patterns (specifically, the MD5 hash of the comma-separated list of patterns; see file.rb). I suspect Logstash 2.0 might improve this situation as the roadmap talks about monitoring improvements.

See Logstash parsing progress bar for an example of how to continuously track Logstash's progress.