0

I am new to ELK stack. Stuck with data ingestion in elasticsearch using logstash.

I am mentioning the steps I followed:- Installed ELK Stack successfully. Then installed plugin logstash-input-mongodb. After that configured logstash with below file:-

input {
    mongodb {
        uri => 'mongodb://localhost:27017/dbName'
    placeholder_db_dir => '/opt/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'notifications'
    batch_size => 1
    }
}

output {
    elasticsearch {
    action => "index"
        index => "notifications_data"
        hosts => ["localhost:9200"]
    }
    stdout { codec => json }
}

Saved the above file as mongo-connector.conf

then run this using /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/mongo-connector.conf

After this, the logs on the terminal was:-

D, [2020-11-07T14:01:45.739178 #29918] DEBUG -- : MONGODB | localhost:27017 req:480 conn:1:1 sconn:78 | dbName.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"dbName", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x4508feda2dce4ec6...>}}
D, [2020-11-07T14:01:45.741919 #29918] DEBUG -- : MONGODB | localhost:27017 req:480 | dbName.listCollections | SUCCEEDED | 0.002s
D, [2020-11-07T14:01:50.756430 #29918] DEBUG -- : MONGODB | localhost:27017 req:481 conn:1:1 sconn:78 | dbName.find | STARTED | {"find"=>"notifications", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5fa012440d0e947dd8dfd2f9')}}, "limit"=>1, "$db"=>"dbName", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x4508feda2dce4ec6...>}}
D, [2020-11-07T14:01:50.758080 #29918] DEBUG -- : MONGODB | localhost:27017 req:481 | dbName.find | SUCCEEDED | 0.001s
D, [2020-11-07T14:01:50.780259 #29918] DEBUG -- : MONGODB | localhost:27017 req:482 conn:1:1 sconn:78 | dbName.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"dbName", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x4508feda2dce4ec6...>}}
D, [2020-11-07T14:01:50.782687 #29918] DEBUG -- : MONGODB | localhost:27017 req:482 | dbName.listCollections | SUCCEEDED | 0.002s
D, [2020-11-07T14:01:53.986862 #29918] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'standalone' to 'standalone' [awaited].
D, [2020-11-07T14:01:53.987784 #29918] DEBUG -- : MONGODB | There was a change in the members of the 'Single' topology.
D, [2020-11-07T14:01:54.311966 #29918] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'standalone' to 'standalone'.
D, [2020-11-07T14:01:54.312747 #29918] DEBUG -- : MONGODB | There was a change in the members of the 'Single' topology.
D, [2020-11-07T14:01:55.799418 #29918] DEBUG -- : MONGODB | localhost:27017 req:483 conn:1:1 sconn:78 | dbName.find | STARTED | {"find"=>"notifications", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5fa012440d0e947dd8dfd2f9')}}, "limit"=>1, "$db"=>"dbName", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x4508feda2dce4ec6...>}}

Below is the logstash logs file:-

[2020-11-07T16:32:33,678][WARN ][logstash.inputs.mongodb  ][main][6a52e3ca90ba4ebc63108d49c11fcede25b196c679f313b40b02a8e17606c977] MongoDB Input threw an exception, restarting {:exception=>#<Sequel::DatabaseError: Java::JavaSql::SQLException: attempt to write a readonly database>}

Index gets created on elasticsearch but docs don't get inserted there.

Himanshu Dhingra
  • 317
  • 3
  • 7
  • did you check dead_letter_queue? – hamid bayat Nov 07 '20 at 09:27
  • @hamid Just checked about it but I have tried the above example with different data sets or different collections. So do we need to try dead_letter_queue? – Himanshu Dhingra Nov 07 '20 at 10:32
  • i meant it is possible the data is rejected and saved in dead_letter_queue. check the path and see is it empty? if it is has data, try read the cause. – hamid bayat Nov 07 '20 at 10:34
  • No logs found in dead_letter_queue but found logstash logs ```[2020-11-07T16:32:33,678][WARN ][logstash.inputs.mongodb ][main][6a52e3ca90ba4ebc63108d49c11fcede25b196c679f313b40b02a8e17606c977] MongoDB Input threw an exception, restarting {:exception=>#} ``` – Himanshu Dhingra Nov 07 '20 at 11:03
  • Check that the user you are using to run logstash has write permissions on the placeholder directory and placeholder db file. This input plugin will need that to track the progress of what it has already processed from mongodb collection. – karan shah Nov 07 '20 at 19:37
  • @karanshah I have provided that file 777 permission. But still, this DB is restarting again and again. And DB gets updated but only on restarting logstash. – Himanshu Dhingra Nov 07 '20 at 20:25

0 Answers0