2

I basically want to set the prefetch to 1. However I don't think there's a way to do this when running celery worker -prefetch=1 or something like that. I need to create a settings file but not sure what this file should be named as (settings.py?) and where this file should reside.

Do all the celery worker need this settings file?

About the prefetch value I read Understanding celery task prefetching to understand it better.

Community
  • 1
  • 1
user299709
  • 4,922
  • 10
  • 56
  • 88

2 Answers2

3

You can define settings for Celery by creating a module named celeryconfig and setting the appropriate directives there. The module file, celeryconfig.py, has to be available on the Python path. See configuration docs.

You can set the prefetch value using the CELERYD_PREFETCH_MULTIPLIER directive.

famousgarkin
  • 13,687
  • 5
  • 58
  • 74
  • do i create a file called celeryconfig.py and set CELERYD_PREFETCH_MULTIPLIER = 1 in that file and put it in the /usr/bin/python ? – user299709 Sep 14 '14 at 06:56
  • @user299709 Yes, you create a file like this, but you don't copy it to Python bin. To have it on the Python path means to put it in a location that Python looks to when searching for modules and packages. This depends on your Python environment. You can get your Python path value using [`sys.path`](https://docs.python.org/2/library/sys.html#sys.path), e.g. simply `$ python -c 'import sys; print(sys.path)'` and see the output. It will contain various Python directories but it should include either the current directory (empty string `''`) or the directory of the script invoked. – famousgarkin Sep 14 '14 at 07:34
  • do I need to import celeryconfig in my task script? or does celery worker already know about it? can I put celeryconfig.py in the same directory as my task.py (which celery worker knows about) – user299709 Sep 14 '14 at 08:32
  • okay how do i know if the config is imported and working properly? – user299709 Sep 14 '14 at 08:43
2

If you need to change it on the command line, use 'celery worker [options] -- [configuration] 

 celery worker -n slow.task.Worker -A myworker -Q queueA -- celeryd.prefetch_multiplier=1

 celery worker -n Fast.task.Worker -A myworker -Q queueB -- celeryd.prefetch_multiplier=8

This way you can have the same celeryconfig.py but with some difference set in the command line.

ant31
  • 4,471
  • 4
  • 15
  • 20