0

On Python/Django stack, we were used to using Celery along with RabbitMQ. Everything was easily done.

However when we tried doing the same thing in Clojure land, what we could get was Langhour. In our current naive implementation we have a worker system which has three core parts.

  • Publisher module
  • Subscriber module
  • Task module

We can start the system on any node in either publisher or subscriber mode. They are connected to RabbitMQ server.

They share one worker_queue.

What we are doing is creating tasks in Task module, and then when we want to run a task on subscriber. we send an expression call to the method, in EDN format to Subscriber which then decodes this and runs the actual task using eval.

Now is using eval safe ? we are not running expressions generated by user or any third party system.Initially we were planning to use JSON to send the payload message but then EDN gave us a lot more flexibility and it works like a charm, as of now.

Also is there a better way to do this ?

Amogh Talpallikar
  • 12,084
  • 13
  • 79
  • 135
  • The repercussions of using eval in lisplike languages like Clojure has been covered in http://stackoverflow.com/questions/2571401/why-exactly-is-eval-evil – NielsK Nov 11 '13 at 20:27

1 Answers1

1

Depends on you needs (and your team), I highly suggest Storm Project. You will get a distributed, fault tolerant and realtime computation and it is really easy to use.

Another nice thing in Storm that it supports a plethora of options as the datasource for the topologies. It can be for example: Apache Kafka, RabbitMQ, Kestrel, MongoDB. If you aren't satisfied, then you can write your own driver.

It is also has a web interface to see what is happening in your topology.

Chiron
  • 20,081
  • 17
  • 81
  • 133