Logstash not writing data from external file to elasticsearch

Question

I have a sample text file named testfile.txt containing simple "Hi". I want this data to get indexed on ElasticSearch. I run the following command on logstash:

bin/logstash -f logstash-test.conf

Conf File content is below:

input{
    file
        {
            path=> "/home/abhinav/ELK/logstash/testfile.txt"
            type => "test"
            start_position => "beginning"
        }
     }
output{
     elasticsearch { host => localhost 
             index => "test_index"
               }
     stdout { codec => rubydebug } 
      }

The ElasticSearch Log shows the follwing error:

[2015-05-04 14:52:23,082][INFO ][cluster.service ] [Argo] added {[logstash-abhinav-HP-15-Notebook-PC-10919-4006][CPk1djqFRnO-j-DlUMJIzg][abhinav-HP-15-Notebook-PC][inet[/192.168.1.2:9301]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[logstash-abhinav-HP-15-Notebook-PC-10919-4006][CPk1djqFRnO-j-DlUMJIzg][abhinav-HP-15-Notebook-PC][inet[/192.168.1.2:9301]]{client=true, data=false}])

I tried following things:

Tried with simple std input(stdin) to ES n stdout . It worked.

So, you solved your own problem? If so, please post the solution as an answer. — Christopher Bottoms, May 04 '15 at 12:26

score 2 · Accepted Answer · edited May 23 '17 at 10:28

2

If you are using the same file repeatedly to test with, you are running into the "sincedb" problem -- see How to force Logstash to reparse a file?

You need to add sincedb_path => "/dev/null" to your file input. Generally this is not needed in a production scenario, but it is when testing.

edited May 23 '17 at 10:28

Community

1
1

answered May 04 '15 at 13:34

Alcanzar

16,985
6
42
59

Thanks ! You told me the right concept. Though this thing , adding sincedb path , didn't work, but I got things Right. I reinstalled E L K , then for tried running different conf files and its working now. – Abhinav Choudhary May 06 '15 at 04:50

score 0 · Answer 2 · edited May 23 '17 at 11:52

I have found solution to my problem and also found things we can check, if logstash & ES not working/communicating properly. :

Make sure ES & Logstash are properly installed
Versions installed are compatible with each other
Even while testing, try making different Logstash conf file for different test case, as suggested by Mr. Alcanzar in above comment.

You can also refer to below links , for help regarding this issue :

Cannot load index to elasticsearch from external file, using logstash

https://logstash.jira.com/browse/LOGSTASH-1800

Logstash not writing data from external file to elasticsearch

2 Answers2