5

I have following setup:

  • RabbitMQ broker + Django
  • Celery with CELERYD_PREFETCH_MULTIPLIER=32 (I have a lot of small task thus prefetching them makes a lot of sense from performance standpoint)
  • CELERY_ACKS_LATE=False (tasks are not idempotent)

I run celery in docker container, so when I rebuild docker celery workers are not gracefully shut down. This is ok, if tasks are not acknowledged as broker will sent them back once workers be up again in new docker container, but in other case they - will be lost.

In flower admin panel prefetched tasks have status received.

I had carefully read official documentation and related question and intuitively I feel that prefetched tasks in my setup are acknowledged. Is it so?

Community
  • 1
  • 1
Dmytriy Voloshyn
  • 1,032
  • 12
  • 27

1 Answers1

2

With CELERY_ACKS_LATE=False task will be acknowledged as soon as worker starts to execute it: just before execution.

So if a worker starts to execute a task and you kill it task will be lost. Other prefetched tasks won't be lost because they are still unacknowledged.

Raz
  • 7,508
  • 4
  • 27
  • 25
  • Thats sounds reasonable, but prefetched tasks in flower are marked as 'RECEIVED' - does it mean they have not started to to be executed? – Dmytriy Voloshyn Feb 13 '17 at 10:26
  • I guess `RECEIVED` means `Unacked` in terms of Rabbitmq. Other workers can't prefetch them because they are not `Ready`. But if worker loses connection without acknowledgment they'll become `Ready`. – Raz Feb 13 '17 at 10:40
  • But if I'll reboot celery server(hard reboot) how will rabbitmq know that these tasks are able to be assigned to other nodes? – Dmytriy Voloshyn Feb 13 '17 at 10:42
  • May be this [question](http://stackoverflow.com/questions/31915773/rabbimq-what-are-ready-unacked-types-of-messages) will be useful. – Raz Feb 13 '17 at 10:42
  • It is done automatically by rabbitmq when consumer closes its channel. – Raz Feb 13 '17 at 10:46
  • [another useful explanation](http://stackoverflow.com/questions/7063224/how-can-i-recover-unacknowledged-amqp-messages-from-other-channels-than-my-conne) – Raz Feb 13 '17 at 10:47
  • Hi, thanks for your answers, indeed prefetched tasks turned out not to be acknowledged - they return to ready state as soon as celery workers die. Also we found a way to restart docker containers in more graceful way and now everything works fine. – Dmytriy Voloshyn Feb 15 '17 at 08:32