Celery: list all tasks, scheduled, active and finished

Question

Update for the bounty

I'd like a solution that does not involve a monitoring thread, if possible.

I know I can view scheduled and active tasks using the Inspect class of my apps Control.

i = myapp.control.inspect()

currently_running = i.active()
scheduled = i.scheduled()

But I could not find any function to show already finished tasks. I know that this information mus be at least temporarily accessible, because I can look up a finished task by its task_id:

>>> r = my task.AsyncResult(task_id=' ... ')
>>> r.state
u'SUCCESS'

How can I get a complete list of scheduled, active and finished tasks? Or possibly a list of all tasks at once?

I agree that the information is retrivable. I use flower and the flower application can retrive the state of all the tasks. I find that there is a cache dictionary in the backend, but only stores a few tasks `myapp.backend._cache`. Flower has a restful api, I think that you can use that in your web monitoring application, but It must be another way... — Pablo Navarro, Sep 14 '12 at 20:53

score 10 · Answer 1 · edited May 01 '19 at 08:31

10

Celery Flower shows tasks (active, finished, reserved, etc) in real time. It enables to filter tasks by time, workers and types.

https://github.com/mher/flower

edited May 01 '19 at 08:31

ZeWaren

3,978
2
20
21

answered Sep 07 '12 at 07:45

mher

10,508
2
35
27

3

I'm required to integrate a monitoring solution within a website, so unfortunately I cannot use an out-of-the box solution like flower. But I'll look into it anyways, since it supports my assumption, that the information is indeed retrievable. So thanks for the hint. +1 – Constantinius Sep 07 '12 at 07:49
Check out how it is implemented in flower or use flower api – mher Sep 07 '12 at 08:06
I am trying to solve puzzle of where tasks info is stored... I can browse 2000 completed tasks in Flower but the redis dbs configured as broker and result backend are both empty (no keys)... I thought I should see something there? – Anentropic Mar 13 '13 at 14:28
4

Flower stores them in memory - if you restart it, all (at least all finished) tasks are gone. – ogurets Mar 24 '16 at 00:12

score 2 · Answer 2 · edited May 23 '17 at 12:08

One option not requiring a monitoring thread is a Celery on_success handler (using bootsteps feature in 3.1+) - this would need to write relevant info to your own datastore.

You need to create a custom task class to do this. This on_failure example gives an idea.

Possibly better option, needing less code, is to use a task_success signal in a similar way, recording the info you need later.

The Flower option is probably simpler, as you are querying info already maintained by Flower when tasks complete - see this answer.

Celery: list all tasks, scheduled, active and finished

2 Answers2

Linked

Celery: list all tasks, scheduled, active *and* finished

2 Answers2

Linked

Celery: list all tasks, scheduled, active and finished