0

I need to develop a platform in Java to download tweets from Twitter (that was obvious). The idea is to have various computers downloading from the streaming API and a main controller to send tasks (keywords to download and other data) to each fetcher. My problem is related with the connection between this programs. Which is the best way to do this? Actually I'm using RMI to send commands like "stop", "start", "setTask" from the Controller (client) to each fetcher (servers) and a SSLSocket to make a quick validation, but I'm not sure if this is a good idea. I could use TCP sockets but maybe it's not a good idea to have permanent connections. What do you think? Is it a good idea to keep using RMI or should I take another point of view?

Thank you ;)

David Moreno García
  • 4,423
  • 8
  • 49
  • 82

1 Answers1

1

I propose you to use queue (and any queue protocol). ActiveMQ, RabbitMQ, QPID, or one of many other tool.

I use ActiveMQ in prod and fine with it, but for very highroad RabbitMQ will be better.

you receive easy scaling for any count of workers and easiest way to share/split tasks between workers.

Also please look on ActiveMQ or RabbitMQ or ZeroMQ or

Community
  • 1
  • 1
iMysak
  • 2,170
  • 1
  • 22
  • 35
  • There wouldn't be so many messages. Only a message to set the task, other to start the fetcher and a stop message to stop it. Having said this, I'm not sure if is a good idea to use this queues. – David Moreno García Mar 04 '13 at 22:59
  • maybe, but as for me - doesn't matter how much tasks you have. Queue strategy is really simple and contain a lot of cool features inside: you newer miss the task is worker will be down, you can launch 2 and more workers and they will be work in parallel, and you don't need to write addition code for it. If you don't like use so big tool — use Http, REST/JSON – iMysak Mar 05 '13 at 00:31