logstash simple file input/output

Question

I have trouble getting logstash to work. The Basic logstash Example works. But then I struggle with the Advanced Pipeline Example. Perhaps it could be as well a problem with elasticsearch.

Now I just want to check if a simple example work:

input: read textfile-a
output: generate new textfile-b with input of the textfile-a

But I am struggling with that. My config is the following:

# foo.conf
input {
    file {
        path => "C:/logstash-2.3.1/logstash-tutorial-dataset"
        start_position => "beginning"
    }
}
output {
    stdout {}
    file {
        #message_format => "%{foo},%{bar},%{fii},%{bor},%{bing}" 
        #codec => { line { format => "custom format: %{message}"}}
        path => "C:/output.txt"
    }
}

When I run logstash, I get the following response and nothings happens.

bin/logstash -f foo.conf -v --debug --verbose
io/console not supported; tty will not be manipulated
{:timestamp=>"2016-04-22T13:41:15.514000+0200", :message=>"starting agent", :level=>:info}
{:timestamp=>"2016-04-22T13:41:15.518000+0200", :message=>"starting pipeline", :id=>"main", :level=>:info}
{:timestamp=>"2016-04-22T13:41:16.035000+0200", :message=>"Registering file input", :path=>["C:/logstash-2.3.1/logstash-tutorial-dataset"], :level=>:info}
{:timestamp=>"2016-04-22T13:41:16.039000+0200", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"c:/Users/foobar/.sincedb_802dc9c88c8fad631bf3d3a5c96435e4", :path=>["C:/logstash-2.3.1/logstash-tutorial-dataset"], :level=>:info}
{:timestamp=>"2016-04-22T13:41:16.103000+0200", :message=>"Starting pipeline", :id=>"main", :pipeline_workers=>4, :batch_size=>125, :batch_delay=>5, :max_inflight=>500, :level=>:info}
{:timestamp=>"2016-04-22T13:41:16.106000+0200", :message=>"Pipeline main started"}

how do I get the simple example working?

score 3 · Accepted Answer · answered Oct 17 '16 at 16:23

ignore_older => 0 did the trick, see documentaion: ignore_older.

The working configuration is the following:

# foo.conf
input {
    file {
        path => "C:/logstash-2.3.1/logstash-tutorial-dataset"
        start_position => "beginning"
        ignore_older => 0  
    }
}
output {
    stdout {}
    file {
        path => "C:/output.txt"
    }
}

Now the .sincedb* file contains as well content.

score 1 · Answer 2 · answered Apr 22 '16 at 15:50

1

Logstash remembers which files it has processed, and how much of them it has processed. In normal operations, this allows it to restart in case of failure and not reprocess logs.

In your case, I imagine that your log file has been processed once already, so logstash is ignoring it. The "start_position" parameter you've provided is documented to only apply to new files.

You would either need to reset your registry (perhaps files like /var/lib/logstash/.sincedb*), or set the "sincedb_path" parameter in your file{} into to /dev/null so that it doesn't maintain the history while you're testing.

answered Apr 22 '16 at 15:50

Alain Collins

16,268
2
32
55

The sincedb is empty. I removed it several times, but there was no change in the result. Something is wired. Is there another simple hello world test, thst i can run for test? – jerik Apr 26 '16 at 19:57
If the file is empty, then logstash isn't processing your file. Perhaps it's too old now (see "ignore_older")? – Alain Collins Apr 27 '16 at 02:57

logstash simple file input/output

2 Answers2