5

Does celery support returning pending tasks number before given task id?

For example, without celery worker started, I push task1, task2, task3, all the three are pending, now, what I wanna is, if I give task3, it tells me there are 2 pending tasks before 3.

I use celery celery 4.1, rabbitmq 3.5.4 as broker, and redis 3.2.9 as result backend.

Although I can get rabbit queue depth by management API(e.g. get_queue_depth from pyrabbit package), this results the whole queue depth, not pending number before specified task id.

And I know I could maintain a queue managing pushed task ids by myself.

But I wanna if there is any easy way by celery or rabbitmq itself.

Thanks.

Storm
  • 53
  • 3

1 Answers1

1

I'm not sure if it answer your question but there is control client that can help you to inspect reserved tasks, active tasks and so on..

i = app.control.inspect()
i.reserved()

#output:
[{'worker1.example.com':
    [{'name': 'tasks.sleeptask',
      'id': '32666e9b-809c-41fa-8e93-5ae0c80afbbf',
      'args': '(8,)',
      'kwargs': '{}'}]}]

for more info: http://docs.celeryproject.org/en/latest/userguide/workers.html#dump-of-reserved-tasks

You can also monitor/inspect from command line: http://docs.celeryproject.org/en/latest/userguide/monitoring.html#commands

ItayB
  • 10,377
  • 9
  • 50
  • 77
  • Although tasks in i.reserved() is list, but sequence is not exactly as the order be executed. So, this shouldn't work. – Wesley Sep 20 '17 at 11:44
  • @Wesley did u try? – ItayB Sep 20 '17 at 13:56
  • not yet... I think you wanna disable message prefetching, I need to find way to close this guy, seems set CELERYD_PREFETCH_MULTIPLIER = 1 is not enough, but first comes a problem that, set this guy to 1 will decrease performance. Don't know if there is any other way not decreasing performance – Wesley Sep 20 '17 at 15:27
  • I'm working with celery in high load and set this one (for using priority queues) - it shouldn't be a problem. BTW you can still increase the worker concurrency.. – ItayB Sep 20 '17 at 15:31
  • https://stackoverflow.com/questions/16040039/understanding-celery-task-prefetching this post explain a lot about prefetching. However, we make a fatal mistake till now...that is, inspect is only for running celery workers, not for rabbitmq. We cannot get the pending tasks within rabbitmq from celery inspect. I think we have to solve the issue from the mq side. I am thinking about get_messages of pyrabbit package. – Wesley Sep 21 '17 at 06:34
  • can u share what is your goal? why do you need that info? – ItayB Sep 21 '17 at 07:50