8

I'm running multiple Django/apache/wsgi websites on the same server using apache2 virtual servers. And I would like to use celery, but if I start celeryd for multiple websites, all the websites will use the configuration (logs, DB, etc) of the last celeryd instance I started.

Is there a way to use multiple Celeryd (one for each website) or one Celeryd for all of them? Seems like it should be doable, but I can't find out how.

Andrei
  • 10,918
  • 12
  • 76
  • 110
Tickon
  • 1,058
  • 1
  • 16
  • 25
  • 2
    wouldn't having different broker_user, broker_password in your celery settings for each website help? – Crazyshezy Dec 18 '12 at 07:51
  • Yes! All I needed was a different broker_user and broker_vhost. Thanks! – Tickon Dec 18 '12 at 19:36
  • @Crazyshezy, For queueing the tasks, yes, but the daemon also needs to be pointed to the correct Django settings file in order to execute it in the right environment. I think the OP's asking how to do the later. If you're running only a single daemon, there's no easy way to do that. frog32's answer is probably the best solution. – Cerin Jun 11 '13 at 16:08

2 Answers2

8

This problem was a big headache, i didn't noticed @Crazyshezy 's comment when i first came here. I just accomplished this by changing Broker URL in settings.py for each web app.

app1.settings.py

BROKER_URL = 'redis://localhost:6379/0'

app2.settings.py

BROKER_URL = 'redis://localhost:6379/1'
Ryu_hayabusa
  • 3,666
  • 2
  • 29
  • 32
  • do you run them under one supervisor instance? when i start my supervisor, it uses the same settings file for each app (even though i use different manage.py files in the 'command' part of supervisord.conf), so my different BROKER_URL settings have no effect. – Felix Böhme Nov 03 '14 at 16:45
  • 1
    @FelixBöhme I am not running this using supervisor. can't you just specify settings using "manage.py celery worker --settings=app.settingsx". – Ryu_hayabusa Nov 03 '14 at 18:06
  • ah, good idea. thanks, that would have been easier. saw this too late, unfortunately. i ended up taking it out of django-supervisor and just using supervisord directly. but at least i know i can go back to using django-supervisor if i need. – Felix Böhme Nov 11 '14 at 01:55
3

Yes there is a way. We use supervisor to start celery daemons for every project we need it.

The supervisor config file looks something like this:

[program:PROJECTNAME]
command=python manage.py celeryd --loglevel=INFO --beat
environment=PATH=/home/www-data/projects/PROJECTNAME/env/bin:/usr/bin:/bin
directory=/home/www-data/projects/PROJECTNAME/
user=www-data
numprocs=1
umask=022
stdout_logfile=/home/www-data/logs/%(program_name)s.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/home/www-data/logs/%(program_name)s.error.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10
autorestart=true
autostart=True
startsecs=10
stopwaitsecs = 60
priority=998

There is also an other advantage if you use this setup: The celery daemons run entirely in userspace.

Remember to use different broker backends for your projects. It won't work if you use the same rabbitmq virtualhost or if you use the same redis database for every project.

frog32
  • 966
  • 7
  • 19
  • Thanks, I got it to work just by using different Rabbitmq user and vhost. But would there be advantages of using supervisor? – Tickon Dec 18 '12 at 21:14
  • It provides us a clean split in responsibilities. The server administrator has to make sure supervisor is always running. We only have to mess with our workers and don't need root privileges to do this. – frog32 Dec 19 '12 at 08:45
  • @frog32 how would this work if the project versions are released within timestamped dirs (in this case apache points to a wsgi file symlink which is updated by the deployment script every release, so there's no real way of telling the absolute dir to the virtualenv, unless we create another symlink). – user2298943 Aug 06 '14 at 16:09
  • 1
    @user2298943 There shouldn't be a problem using a symlink as directory. To be safe you should first stop the task then change the symlink and after that start the task again. – frog32 Aug 06 '14 at 19:50