10

I'm running a site using Django 10, RabbitMQ, and Celery 4 on CentOS 7.

My Celery Beat and Celery Worker instances are controlled by supervisor and I'm using the django celery database scheduler.

I've scheduled a cron style task using the cronsheduler in Django-admin.

When I start celery beat and worker instances the job fires as expected.

But if a change the schedule time in Django-admin then the changes are not picked up unless I restart the celery-beat instance.

Is there something I am missing or do I need to write my own scheduler?

Celery Beat, with the 'django_celery_beat.schedulers.DatabaseScheduler' loads the schedule from the database. According to the following doc https://media.readthedocs.org/pdf/django-celery-beat/latest/django-celery-beat.pdf this should force Celery Beat to reload:

A schedule that runs at a specific interval (e.g. every 5 seconds). • django_celery_beat.models.CrontabSchedule A schedule with fields like entries in cron: minute hour day-of-week day_of_month month_of_year.

django_celery_beat.models.PeriodicTasks This model is only used as an index to keep track of when the schedule has changed. Whenever you update a PeriodicTask a counter in this table is also incremented, which tells the celery beat service to reload the schedule from the database. If you update periodic tasks in bulk, you will need to update the counter manually:

from django_celery_beat.models import PeriodicTasks
PeriodicTasks.changed()

From the above I would expect the Celery Beat process to check the table regularly for any changes.

Steve
  • 141
  • 2
  • 14
  • 1
    possible duplicate of: http://stackoverflow.com/questions/21666229/celery-auto-reload-on-any-changes – matias elgart Nov 13 '16 at 23:05
  • Don't think it is a duplicate as I want Celery Beat to detect when the schedule has been updated in the database. – Steve Nov 13 '16 at 23:46
  • 1
    Definitely, not a duplicated. The linked question is talking about reload celery worker if Django settings change (a static `.py` file), not about `celery beat` and reload when database tasks change. – Ruben Nov 15 '16 at 17:48
  • Looks like [this Github issue](https://github.com/celery/django-celery-beat/issues/7) is a discussion of the same problem: (they reference this post). Some solutions proposed, none simple. Also [here](https://github.com/celery/celery/issues/3493). No solution given. – Gabriel May 21 '17 at 08:13

2 Answers2

2

i have changed the celery from 4.0 to 3.1.25, django to 1.9.11 and installed djcelery 3.1.17. Then test again, It's OK. So, maybe it's a bug.

kenda zheng
  • 21
  • 1
1

I have a solution by:

  1. Creating a separate worker process that consumes a RabbitMQ queue.
  2. When Django updates the database it posts a message to the queue containing the name of the Celery Beat process (name defined by Supervisor configuration).
  3. The worker process then restarts the named Celery Beat process.

A bit long winded but does the job. Also makes it easier to manage multiple Django apps on the same server that require the same functionality.

Steve
  • 141
  • 2
  • 14
  • You mean, run separate process like: `celery beat ...` and `celery worker ...`? I already doing this and still not working. – Ruben Nov 16 '16 at 09:07
  • Yes, using the database scheduler. Whenever the settings (time etc) change in the database you need to restart the Celery Beat process so that it reads the database again. – Steve Nov 16 '16 at 09:43
  • 1
    If you have to restart Celery Beat process manually you are not solving the problem, aren't you? "But if a change the schedule time in Django-admin then the changes are not picked up unless I restart the celery-beat instance." – Ruben Nov 16 '16 at 11:08
  • You need to add a signal when the record is saved to post a message to restart Celery Beat. – Steve Nov 16 '16 at 11:38
  • Whilst it's not fully solving the problem as such, it's a solution that works for now. I'm not sure how the Celery Beat process could detect changes to the database? FYI the database is Postgresql. – Steve Nov 16 '16 at 11:45
  • 1
    Ah OK, I missed the signal part to auto-reload Celery Beat. It is suppose to be auto-reload when something changes on DB reading `PeriodicTasks` but it doesn't work. After debugging the library, I agree with @kenda-zheng, it sames a bug. Finally, I downgrade the project as @kenda-zheng response and it works. – Ruben Nov 16 '16 at 12:22