4

I have a series of servers running multiple workers. These are long-running tasks, requiring anywhere from 10 minutes to 36 hours. As such, I'd like to avoid prefetching if at all possible and have each worker pick up a singular new task after completing.

I'm using the celeryd init.d service, and have 

CELERYD_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1

in /etc/default/celeryd of the worker server.

However, if I go to the python shell console and do 

from work_project.celery import app
inspector = app.control.inspect()
inspector.stats()

I get this in the dict output:

...
u'prefetch_count': 4,
...

I'm using RabbitMQ as the broker and Redis as the backend. I suspect this prefetching is having the effect of several workers remaining idle after completing their initial task because other workers have pending tasks in their queues. For example, currently I have two servers with nine workers total running. At the start of a 20 task batch, all nine were running concurrently. Now, approximately 90m later, only six workers are active.

Jason
  • 11,263
  • 21
  • 87
  • 181
  • Any updates on this @Jason ..I'm facing the same issue – SomeTypeFoo Feb 21 '17 at 10:08
  • 1
    I found adding `--maxtasksperchild=1' to the `CELERY_OPTS` parameter in `/etc/defaults/celeryd` with the goal of having the worker execute one task and then terminating itself did the trick for this particular issue. – Jason Feb 21 '17 at 11:16
  • Thanks so much Jason for sharing. One more thing, did more of a googling - This link should help anyone ending up here which solved for me: [SO post](http://stackoverflow.com/questions/16040039/understanding-celery-task-prefetching) and this [Celery DOCS](http://docs.celeryproject.org/en/latest/userguide/optimizing.html#worker-settings) – SomeTypeFoo Feb 21 '17 at 11:23

1 Answers1

2

I deal with the same problem this way.

Running worker with: celery worker --prefetch-multiplier=1 --concurrency=1 -Ofair

Under Django settings: CELERY_TASK_ACKS_LATE = True

Django==3.0.2
celery==4.4.0

It's important to have concurrency=1, because prefetch-multiplier multiplies on concurrency

-Ofair - might be redundant, need to check

Acks late param on worker didn't find (or didn't work) for me

Setting without django need to check (too lazy for it). Might be CELERYD_TASK_ACKS_LATE = True

Snobby
  • 1,067
  • 3
  • 18
  • 38