Celery: number of workers vs concurrency

Question

What is the difference between having:

for the same queue.

Thanks

score 20 · Answer 1 · answered Feb 27 '20 at 09:25

20

I assume that you are running both workers in the same machine. In that case I would recommend you to maintain one worker for a queue.

Two workers for the same queue does not benefit you by any means. It would just increase the memory wastage.
Two or more workers when you have multiple queues, to maintain priority or to allocate different number of cores to each worker.
Two or more workers for a single queue is useful if you run the workers in different machines. The workers in different machines consumes tasks from same queue, you could allocate concurrency based on the cores available in each machine.

I do realise I responded 2+ years later. But I just thought I'll put it here for anyone who still has similar doubts.

answered Feb 27 '20 at 09:25

Aditya Nagesh

1

Any more details on the first bullet? I have two workers consuming the same queue, and it seems to work just fine, what exactly would be the downside? Or why there would be no upside? Why would the queue be concerned about how many consumers it has? – lowercase00 Nov 01 '21 at 22:18
I think they mean two workers on the same *core*, which would mean the workers would use context-switches in python, like the fake threads. But even then, it would speed up any network related layers the tasks has since they have to wait in tcp. So their first point is simply incorrect unless I'm missing something else. – Işık Kaplan Apr 21 '22 at 00:43
@IşıkKaplan I agree with you. I've also found adding more workers speeds up the number of tasks being processed. I am using 4 workers each set to eventlet. Additionally [Celery documentation](https://docs.celeryq.dev/en/latest/userguide/workers.html#concurrency) says the following `There’s even some evidence to support that having multiple worker instances running, may perform better than having a single worker.` – Quazi Irfan Jun 21 '22 at 04:48

ItayB · Accepted Answer · 2017-10-03T13:57:19.287

Intersting question.

Things that I can think of (I'm sure there are a lot more):

For having high availability:
1. You want more than one machine (if one goes down) - so you must use worker per machine.
2. Even for one machine - I think it is safer to have 2 workers which run in a two different processes instead of one worker with high concurrency (correct me if I wrong, but I think it is implemented with threads).
In docs I see that they the recommendation is to use concurrency per CPUs.
If you want to separate different tasks to different workers..

Of course, you have price for that: more processes that takes more resources (CPU/Memory etc).

I found this question which is quite similar.

2 Answers2