Running celery task on Ubuntu with supervisor

Question

I have a test Django site using a mod_wsgi daemon process, and have set up a simple Celery task to email a contact form, and have installed supervisor.

I know that the Django code is correct.

The problem I'm having is that when I submit the form, I am only getting one message - the first one. Subsequent completions of the contact form do not send any message at all.

On my server, I have another test site with a configured supervisor task running which uses the Django server (ie it's not using mod_wsgi). Both of my tasks are running fine if I do

sudo supervisorctl status

Here is my conf file for the task I've described above which is saved at

/etc/supervisor/conf.d

the user in this instance is called myuser

[program:test_project]
command=/home/myuser/virtualenvs/test_project_env/bin/celery -A test_project worker --loglevel=INFO --concurrency=10 -n worker2@%%h
directory=/home/myuser/djangoprojects/test_project
user=myuser
numprocs=1
stdout_logfile=/var/log/celery/test_project.out.log
stderr_logfile=/var/log/celery/test_project.err.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

stopasgroup=true

; Set Celery priority higher than default (999)
; so, if rabbitmq is supervised, it will start first.
priority=1000

My other test site has this set as the command - note worker1@%%h

command=/home/myuser/virtualenvs/another_test_project_env/bin/celery -A another_test_project worker --loglevel=INFO --concurrency=10 -n worker1@%%h

I'm obviously doing something wrong in that my form is only submitted. If I look at the out.log file referred to above, I only see the first task, nothing is visible for the other form submissions.

Many thanks in advance.

UPDATE

I submitted the first form at 8.32 am (GMT) which was received, and then as described above, another one shortly thereafter for which a task was not created. Just after finishing the question, I submitted the form again at 9.15, and for this a task was created and the message received! I then submitted the form again, but no task was created again. Hope this helps!

Answer 1

use ps auxf|grep celery to see how many worker you started,if there is any other worker you start before and you don't kill it ,the worker you create before will consume the task,result in you every two or three(or more) times there is only one task is received.

and you need to stop celery by:

sudo supervisorctl -c /etc/supervisord/supervisord.conf stop all

everytime, and set this in supervisord.conf:

stopasgroup=true             ; send stop signal to the UNIX process group (default false)

Otherwise it will causes memory leaks and regular task loss.

---

If you has multi django site,here is a demo support by RabbitMQ: you need add rabbitmq vhost and set user to vhost:

sudo rabbitmqctl add_vhost {vhost_name}
sudo rabbitmqctl set_permissions -p {vhost_name} {username} ".*" ".*" ".*"

and different site use different vhost(but can use same user). add this to your django settings.py:

BROKER_URL = 'amqp://username:password@localhost:5672/vhost_name'

some info here:

Using celeryd as a daemon with multiple django apps?

Running multiple Django Celery websites on same server

Run Multiple Django Apps With Celery On One Server With Rabbitmq VHosts

Run Multiple Django Apps With Celery On One Server With Rabbitmq VHosts