Log rotation in logstash

Question

I am using file as input for logs in logstash . My log files are rotated daily so , I wanted to ask how can we configure file plugin of logstash so that it work with the files that are rotated daily. And adding to this, is log rotation available with file beat as well.

Answer 1

I am trying to answer your questions in part.

First - log rotation.

From the docs:

Note that the rotated filename will be treated as a new file so if start_position is set to beginning the rotated file will be reprocessed.

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html

That means, that if you have a rename in your file rotation, you will likely double your file (unless the path excludes the renamed file I believe).

If your path excludes your renamed file, then it should be fine.

I fixed this in a different way (in java and python accordingly).

I disable renaming of files and instead name the log file with the date prefix. So for me, in my java app, the file name is:

my-server-log-%h-%d.log

Since I am working in a distributed environment, I incorporate the hostname into my logfile name.

%h = hostname %d = date

This ends up in my file being named:

my-server-log-pandaadb-2016-06-20.log

This file is never renamed. I modified my rotation algorithm to simply not rename and instead at midnight create a new file and leave the previous file untouched. This has the effect that logstash (correctly) knows that it has read all lines in the previous file. It picks up the new file since I am using wildcards in my input. No logs are duplicated.

This also works quite well in combination with rsync by the way.

I hope that helps,

Artur

Edit: I have not worked with filebeat so far, so I can't comment on that part.