1

I have tons of data present in MySQL in form of different database, and their respective tables. They all are related to each other. But when I have to do analysis in data, I have to create different scripts, that combine data, merge it and show me as a result, but this takes a lot of time, and effort too. I love elasticsearch for its speed and visualization of data via kibana, therefore I have decided to move my entire MySQL data in real time to elasticsearch, keeping data in MySQL too. But I want a scalable strategy, and process that migrates that data to elasticsearch.

Suggest the best tool, or methods to do the job.

Thank you.

Zeeshan
  • 98
  • 1
  • 4
  • 12
  • 1
    There are a few answers that might help: [here](http://stackoverflow.com/questions/33028085/how-to-migrate-mysql-data-to-elasticsearch-realtime/33029458#33029458), [here](http://stackoverflow.com/questions/33293965/how-to-sync-a-mysql-database-to-external-data-source/33325963#33325963) and [here](http://stackoverflow.com/questions/34477095/elasticsearch-replication-of-other-system-data/34477639#34477639). – Val May 22 '16 at 05:59

1 Answers1

0

  1. Prior to Elasticsearch 2.x you could write your own Elasticsearch _river plugin that you can install into elasticsearch. You can control how often you want this said data you've munged with your scripts to be pulled in by the _river (Note: this is not truly recommended).

  2. You may also use your favourite Queuing Message Broker tool such as ActiveMQ to push your data into elasticsearch

  3. If you want full control to meet your need for real time migration of data you may also write a simple app that makes use of elasticsearch REST end point, by simply writing to it via REST. You can even do bulk POST

  4. Make use of any of the elasticsearch tools such as beat, logstash that are great at shipping almost any type of data into elasticsearch

For other alternatives of munging your data to a flat file, or if you want to maintain relationships see this post here

DaddyMoe
  • 990
  • 1
  • 14
  • 20