1

We would like to configure Logstash so that after it finishes processing an input file, the file is deleted. Is there a way to do this?

Thanks

user1052610
  • 4,440
  • 13
  • 50
  • 101

1 Answers1

1

The stock file input doesn't have a "done" concept. It assumes that a file can always receive more data and never gives up the wait. What you can do is use the stdin input which reads until EOF and then terminates Logstash, after which you can delete the file (if Logstash terminated successfully). So, basically like this:

/path/to/logstash -f /path/to/configfile < logfile.log && rm logfile.log

Unfortunately, if Logstash is shut down while the file is being processed, it's not possible to restart Logstash and have it continue where it left off. If you need to be able to do that, consider using the file input and reading the sincedb files to figure out if Logstash has reached the end of the file. See Understanding sincedb files from Logstash file input for the format of those files.

Community
  • 1
  • 1
Magnus Bäck
  • 11,381
  • 3
  • 47
  • 59
  • This works, but the issue is that if logstash crashes in the middle of processing the file, it will not know that part of the file has already been processed, so duplicate entities will be sent to elasticsearch. Is there a way around that? Thanks – user1052610 Jun 29 '15 at 08:14
  • @user1052610: I've added a paragraph about that. – Magnus Bäck Jun 29 '15 at 10:31
  • David, thanks for clarifying that. The blog is great, and shows how to copy from one index to another with Logstash. I'm not sure what you mean by your reference to the file input plugin in the blog. What we would really like to do is to configure Logstash so that it can use the file input plugin and, on EOF, either 1) rename\move the file, or 2) for Logstash to terminate. Can this be done, and if so, how? – user1052610 Jun 29 '15 at 13:53
  • David? As I explained, Logstash has no such built-in functionality but as I explained you can build it yourself. – Magnus Bäck Jun 30 '15 at 05:44
  • Ok, will look deeper into the sincedb file structure. They appear quite cryptic, but with the reference you sent perhaps it is possible to write a routine to figure out when logstash is done with the file. Thanks – user1052610 Jun 30 '15 at 06:02