1

I have been playing/working with Celery for quite some time now, and as a "Please run this task and retry it a few times if it fails, send me an email if it fails for than 10 times" tool it's pretty awesome. It does exactly that.

The problem is that I'm assigned with a problem that requires me to be able to run several groups of tasks, pause/resume them and stop them.

I know that Celery has queues and that I can call add_consumer and cancel_consumer, but I have noticed that even if I cancel a consumer, the worker will still run all the tasks that we're already prefetched. Which means that there is no way of issuing a Pause *NOW* command. There is just a Pause sometime later, when you're done doing stuff.

A workaround for this is to design the tasks itself to check if they should actually run or sleep.

But then another problem shows: ¿How many tasks do I actually have? Celery will expose a primitive way of querying for active and scheduled tasks, but I have noticed that the tasks that are in RETRY state are neither in the active nor in the scheduled list, which means that querying the total amount of tasks with Celery is error prone.

A workaround for that is to use Redis (or any other kind of database) and store the state of each task; then query the database directly, instead of querying Celery.

What I'm trying to say is that Celery, as far as my experience with it goes, doesn't actually allow any sort of fine-grained control. It just run tasks blindly.

Isn't there something that would allow me to control Celery in a better way? Something that would let me pause/resume/stop entire queues of tasks and maybe be show me some stats (how many tasks in total in this queue?, how many active in this queue?, how many in retry state in this queue?, etc...)

Just FYI: Yes, I know what Flower is. No, Flower does not provide any of the things that I'm asking for. Flower won't show tasks per queue, nor it will let me pause a specific queue instantly. At best, it will allow me to cancel a consumer, which as I said, will stop executing tasks after the prefetched tasks are done.

alexandernst
  • 14,352
  • 22
  • 97
  • 197
  • I use Celery with RabbitMQ . To query stats relating to queue, I need to make API calls to rabbit using RabbitMQ Management HTTP API. – Anil Jan 12 '19 at 17:50
  • @Anil does RabbitMQ in conjunction with Celery support stopping all tasks from a specific queue? – alexandernst Jan 12 '19 at 20:09

1 Answers1

0

As far as I know, Celery does not provide you an ability to pause/resume and stop tasks from the box.

I think, that it is better to first check if the task should be run before the task code. And store this information in Redis.

Controlling abilities of Celery

You can look at:

Artem Rys
  • 149
  • 9