0

We are running a Django site with a Celery worker along with celery beat for cron jobs. On occasion we will have a rogue celery beat process that keeps running when supervisor is restarted, or job in supervisor are restarted (I'm not sure when exactly it happens). Basically what happens is that we end up with two celery beat processes running which duplicates these tasks. This is a problem because one task sends reminder emails and when this happens a lot of our users get duplicate emails.

Does anyone have similar experience with celery beat & supervisor. Is there a way to check that one and only one celery beat process is running at all times?

Dustin
  • 3,965
  • 5
  • 27
  • 33
  • Is this question related? http://stackoverflow.com/questions/9286221/work-around-celerybeat-being-a-single-point-of-failure/12958210#12958210 – TheAxeR Apr 25 '14 at 17:10

1 Answers1

0

Perhaps there is a better answer, but this is what I've come up with so far...

I think the rogue process occurs during our deploy where we restart supervisor. I've gone from this:

fab.sudo('supervisorctl restart all')

to this:

fab.sudo('supervisorctl stop all')
with fab.settings(warn_only=True):
    fab.run("ps auxww | grep 'celery' | awk '{print $2}' | xargs kill -9")
fab.sudo('supervisorctl start all')

This seems to be working, but I'm not sure if this is the most reliable solution.

Dustin
  • 3,965
  • 5
  • 27
  • 33