logstash file input glob not working

Question

I'm starting to explore logstash and this is probably a newbie question, but as far as I have studied this should be working and it isn't.

I have a very simple configuration that just reads log files and dump them to the stdout. It works for a single file and for a list (array) of files, but if I use a glob that matches the same files, nothing happens.

I've tested the glob with a short ruby script and it lists the correct files.

Here is my configuration:

input {
    file {
        path => "/home/lpacheco/*.log"
        start_position => "beginning"
    }
}

output {
    stdout {}
}

If I run this with --verbose I get:

{:timestamp=>"2015-09-23T11:26:47.008000-0300", :message=>"Registering file input", :path=>["/home/lpacheco/*.log"], :level=>:info}
{:timestamp=>"2015-09-23T11:26:47.068000-0300", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"/home/.sincedb_6da9e0c63851aa9d5840ba19efd196cb", :path=>["/home/lpacheco/*.log"], :level=>:info}
{:timestamp=>"2015-09-23T11:26:47.089000-0300", :message=>"Pipeline started", :level=>:info}

Nothing else happens.

I'm using:

logstash 1.5.4
OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)
ruby 1.9.3p484 (2013-11-22 revision 43786) [i686-linux]

score 3 · Accepted Answer · edited May 23 '17 at 10:29

3

You are apparently confronted with a sincedb-issue. Logstash saves the last position of a logfile in a file called sincedb. The sincedb is based on the inode of the log file so that renaming or using globs doesn't have any effect.

Try this input for testing:

input {
    file {
        path => "/home/lpacheco/*.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

From latest docs:

Path of the sincedb database file (keeps track of the current position of monitored log files) that will be written to disk. The default will write sincedb files to some path matching $HOME/.sincedb* NOTE: it must be a file path and not a directory path

For more information, take a look at related questions like this.

edited May 23 '17 at 10:29

Community

1
1

answered Sep 23 '15 at 16:39

hurb

2,177
3
18
32

Great, using "/dev/null" as the sincedb_path worked, as did specifying a file name. What's the problem with ommiting the parameter, though? – lpacheco Sep 23 '15 at 18:37
Ok, you nailed it and it is a sincedb_path issue, but I think that the actual problem is that my log file is not "live", it is a copy from our production server. I thought that using 'start_position => "beginning"' would allow me to keep reusing the file, but it didn't work. 'sincedb_path => "/dev/null"' ignores the sincedb, but is not usefull as an actual configuration. Removing the sincedb file is a better solution, I think. – lpacheco Sep 23 '15 at 18:51
1

You're right. `sincedb_path => "/dev/null"` is for test purpose only. You should not use it in productive environments. In productive case deleting the sincedb files is a better approach in order to reparse the logs. `start_position => "beginning"` is a bit confusing in this situation and you are not the only one who stumbled upon it. – hurb Sep 23 '15 at 19:39
For those using Windows who want to ignore the sincedb_path, point it to `sincedb_path => "nul"` instead. – BenM Jan 09 '17 at 16:17

logstash file input glob not working

1 Answers1