0

My application is interacting with elasticsearch instance for all data retrieval and searchability. Elasticsearch needs to feed data from RestAPI (Bulk data around 2-4 gigs). For that I need to write my rest Client to feed elasticsearch with the rest api. Which is perfectly alright.

Since client database is not exposed and we are integration with Rest endpionts only, how to fetch the last updated data at database (mongoDB). Also there is no field called "Last updated" in mongoDB.

I need to write a custom transporter utility which will run on a fixed time, which will fetch the data from rest apis, compare it with elasticsearch and update the modified content only.

Please help !!!!

Thanks

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Deepak S
  • 11
  • 3
  • Have you considered looking into Logstash and the `http_poller` input + an `elasticsearch` filter? – Val Jul 19 '17 at 13:18
  • Well, I need to get the access token first and then pass this access token with every Rest API call. I can't see a way it can be configured in a way to fulfill my request. Can it ? – Deepak S Jul 21 '17 at 05:31
  • Does this answer help? https://stackoverflow.com/questions/37436376/logstash-http-poller-first-url-requests-response-should-be-input-to-second-url/37438666#37438666 – Val Jul 21 '17 at 05:32
  • It should solve my problem, let me implement it and update you. Thanks Man ! you are the champ – Deepak S Jul 21 '17 at 07:41
  • Hey Val, My Rest APIs are returning me JSON only. Whereas I want Logstash shouldn't change the format of my input JSON. Is there anyway ? – Deepak S Jul 24 '17 at 04:46
  • Hard to tell without knowing what you did so far :-) – Val Jul 24 '17 at 04:49
  • Hey Val, We need to sync bulk data (almost 5 GBs) hence we need multiple worker thread based solution. Do you still suggest to use http-poller ? I am wondering if we can use Queue based solution. something like below - REST API => Producer => RabbitMQ => Multiple consumer = > Elasticsearch. Would you mind sharing your emailId so that i can send you some diagrams and you can review them. – Deepak S Jul 27 '17 at 12:34
  • Feel free to share your diagrams in your above question, other people might be able to give their insights, too. – Val Jul 27 '17 at 12:39
  • Here are the diagram (total#2) Please take a look. [link](http://imgur.com/a/vF8mN). – Deepak S Jul 27 '17 at 13:10
  • Any comments ??? – Deepak S Jul 29 '17 at 04:05

0 Answers0