3

I have a few questions regarding task routing, concurrency and performance. Here is my use case :

I've got one dedicated server to run celery tasks, so I can use all the CPUs to run celery workers on this server.

I have a lot of different python tasks, which I route using : CELERY_ROUTES and because the tasks perform really different types of python code, I created 5 different workers. These worker are created when I deploy my project using ansible, here is an example:

[program:default_queue-celery]
command={{ venv_dir }}/bin/celery worker --app=django_coreapp --loglevel=INFO --concurrency=1 --autoscale=15,10 --queues=default_queue
environment =
    SERVER_TYPE="{{ SERVER_TYPE }}",
    DB_SCHEMA="{{ DB_SCHEMA }}",
    DB_USER="{{ DB_USER }}",
    DB_PASS="{{ DB_PASS }}",
    DB_HOST="{{ DB_HOST }}"
directory={{ git_dir }}
user={{ user }}
group={{ group }}
stdout_logfile={{ log_dir }}/default_queue.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=5
redirect_stderr=true
autostart=true
autorestart=true
startsecs=10
killasgroup=true

I have also a CELERY_QUEUES in settings.py to make the bridge between CELERY_ROUTES and my celery programs (Queues)

CELERY_DEFAULT_QUEUE = 'default_queue'

And if it happens that I don't route a task it will go to my 'default_queue'

To give space to all of my queues, I set --concurrency to 1 for default_queue, and more for my most important queue.

But I am wondering, does AutoScale have impact on the same value as concurrency ? Meaning, if I set concurrency to 1 and --autoscale to 15,10 (example above)

Will my worker 'work' on CPU and process a maximum of 15 tasks on this CPU ? Or does this mean something completely different ?

MiniYuuzhan
  • 108
  • 3
  • 14

1 Answers1

8

It makes no sense setting both concurrency and autoscale since both are means to control the number of worker subprocesses for a given worker instance, as explained here.

--concurrency N means you will have exactly N worker subprocesses for your worker instance (meaning the worker instance can handle N conccurent tasks).

--autoscale max, min means you will have at least min and at most max concurrent worker subprocesses for a given worker instance.

On which CPU each process (the main worker process or any of it's child subprocesses) will run is not predictable, it's an OS thing, but do not assume subprocesses will all run on the same CPU (chances are they won't - that's part of the point of having concurrent subprocesses actually).

bruno desthuilliers
  • 75,974
  • 6
  • 88
  • 118
  • 3
    Thanks for this much needed information. So both of these values will define the number of processus I see when I do 'htop' for example ? And what about CELERYD_CONCURRENCY that can be set in settings.py, is it overridden by autoscale like --concurrency is ? – MiniYuuzhan Jul 05 '17 at 02:08
  • Does this work with eventlet ? My celery executing 1 task at a time even if I set --concurrency 5. – Siddhesh Jan 15 '19 at 07:55
  • 1
    I read about `--autoscale` option will no longer be supported for celery version 4.+. https://github.com/celery/celery/issues/4003. I am migrating to 4.+ from 3.1. Do I need to bump the `--concurrency` values? – alltej May 15 '19 at 19:50