3

The debate I am in currently is whether we should stick with RabbitMQ implementation using Pika or move to celery, what all advantages are there if we go with Celery. From what I have understood is Celery is a distributed job queue that simplifies the management of task distribution. It uses broker (RabbitMQ, Redis and so on) for the sending and receiving the message between client and worker, it also can optionally use backend such as Redis to store the results. Where as RabbitMQ is a message Queue which can be used to perform the Jobs in async manner. Eventually if we use either RabbitMQ and implement it using Pika in python it will do the same Job which is to execute long running processes in background.

The few advantages that I see in using Celery are:

  • Can store result of each task, using backend (such as redis).
  • Easier to implement.
  • Also allows to add retries.
  • Is a distributed job queue, can be run on multiple nodes/clusters.
  • Packages like flower can be used to monitor each task, their states, results, time taken and some other metadata too.
  • Task chaining

But on the other side it seems it does restrict us to use some of the features of RabbitMQ and also it has some limitations like it will connect with broker synchronously (issue on github https://github.com/celery/celery/issues/3884 ) I am familiar with this Question already asked here Why use Celery instead of RabbitMQ? but it does not seem to be clear.

Any help would be highly appreciated.

danish_wani
  • 862
  • 9
  • 14

1 Answers1

1

Erm... it seems to me like you are comparing mosquitoes and elephants here. RabbitMQ+Pika is not a replacement for Celery. However, RabbitMQ+Pika can help you implement a (miniature) service such as Celery, if that is really what you want.

If you use RabbitMQ as backend, Celery (actually kombu) will use something similar to Pika - the celery amqp project, to communicate with the broker.

DejanLekic
  • 18,787
  • 4
  • 46
  • 77
  • Thank you for the answer @dejanlekic From what I understand Celery can use RabbitMQ as a broker and as you rightly said it uses amqp to communicate with the broker. What I am trying to understand is when should one go for Celery and when for RabbitMQ+Pika , what features does celery provide ? And you say it's comparing mosquitoes with elephants, are you referring Celery as elephant ? – danish_wani Jun 28 '22 at 08:19
  • 1
    RabbitMQ+Pika will not run anything for you. You use Pika to communicate with RabbitMQ, and send/receive messages. That is all you get. Celery builds upon this publish/subscribe system to provide reliable, *distributed* task execution and orchestration. Think of RabbitMQ+Pika as low(er) level building block. The only difference, as I mentioned above, is that Celery does not use Pika, it uses the `amqp` instead (which has more/less same functionality as Pika). – DejanLekic Jun 28 '22 at 09:17
  • The advantage points, which I have added in question for celery, does they seem right to you? Can store result of each task, using backend (such as redis). Easier to implement. Also allows to add retries. Is a distributed job queue, can be run on multiple nodes/clusters. Packages like flower can be used to monitor each task, their states, results, time taken and some other metadata too. Task chaining. @dejanlekic – danish_wani Jun 28 '22 at 10:47
  • 1
    I humbly believe it is useless to speak about advantages here as they(RabbitMQ+Pika and Celery) are completely different things... It would be the same as discussing advantage of... QT library over QT Designer (which is built using the QT library!). – DejanLekic Jun 28 '22 at 13:44