Dynamically Schedule Celery Beat task with Flask

Question

I want to be able to let the user of my application start/stop periodic crontab style tasks with Celery beat. Right now I run Celery with

venv/bin/celery worker -A celery_worker.celery --loglevel=info

I got Celery Beat running with this simple example:

@celery.task
def add(x, y):
    return x + y

on my config file:

CELERYBEAT_SCHEDULE = {
    'add-every-30-seconds': {
        'task': 'app.email.add',
        'schedule': timedelta(seconds=30),
        'args': (16, 16)
    },
}
CELERY_TIMEZONE = 'UTC'

then I run the Celery beat worker with

celery -A celery_worker.celery beat -s ~/Documents/cesco-automation/power/celerybeat-schedule

And it works perfectly! But I need to have more control over the schedule.

Also on my app shell, I am able to do this.

>>>add.apply_async([80,800],countdown=30)

>>> from datetime import datetime, timedelta     
>>> tomorrow = datetime.now() + timedelta(days=1)
>>> add.apply_async(args=[10, 10], eta=tomorrow)

Which is great, but I am developing a home-automation app, so I also need to stop the tasks. How do I do this??

I have also found this link that mention about a django custom scheduler classes. Which is exactly what I need. On the Celery docs it mentions the -S flag, but I dont know how to properly add the class to my Flask app. How can I use it with Flask??

Do I really need Celery Beat? Are there other option other the crontab? Crontab seems not to be sharp enough.

The default celery scheduler kinda sucks. Don't even get me started on dealing with timezones with it. I would use a database scheduler, can look at the one for django http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html#using-custom-scheduler-classes — reptilicus, Jun 22 '15 at 16:19
I have mention the same doc on the question, but do not know how to make it work. Can I use the Django scheduler on my Flask app? How? — CESCO, Jun 22 '15 at 17:05
@CESCO, Hi, have you tried using django scheduler in flask app ? if yes, please share — Saad Abdullah, Nov 11 '15 at 09:42
well, i found something...and its working fine...incase anyone need https://github.com/tuomur/celery_sqlalchemy_scheduler — Saad Abdullah, Nov 19 '15 at 08:39
@SaadAbdullah Can you show me how? I keep getting entry = self.Entry(**dict(b[key], name=key, app=self.app)) TypeError: string indices must be integers, not str — CESCO, Nov 19 '15 at 15:22
After some debbugin I realized I am not loading the scheduler class properly. I tried with the S flag and within my config file too. How are you doing this? — CESCO, Nov 19 '15 at 16:18

score 4 · Answer 1 · answered May 21 '16 at 19:30

4

This functionality will be available in Celery ver.4.0. Dynamic task scheduling is only currently supported in the development version : http://docs.celeryproject.org/en/master/userguide/periodic-tasks.html#beat-entries

answered May 21 '16 at 19:30

Jakub Czaplicki

1,787
2
28
50

Looks like Celery 4.0.0 has been released (stable) on 2016-11-04: http://docs.celeryproject.org/en/latest/changelog.html#version-4-0-0 . This means dynamic crontab periodic schedules are available. – Jakub Czaplicki Nov 08 '16 at 12:15
1

do you have an example anywhere how to use dynamic crontabs? – Deusdeorum Mar 16 '17 at 19:52
2

That works only before beat is started. Anything added to the schedule won't be picked up until beat is restarted. There is still no way to add periodic tasks dynamically. – Dineshs91 Dec 06 '17 at 19:56
@Dineshs91, I find this comment and +2 confusing. I'm looking for dynamic celery, too, and there is this [post comment from a year earlier](https://stackoverflow.com/questions/40579804/django-celery-beat-admin-updating-cron-schedule-periodic-task-not-taking-effect#comment68466959_40579804) about the package from Celery peeps, [`django-celery-beat`](https://github.com/celery/django-celery-beat) suggests it's possible to reload the schedule with database change. I'm wondering what specifically you're referring to. – xtian May 05 '20 at 21:05

score 1 · Answer 2 · answered Aug 23 '16 at 16:05

Even though we abandoned flask/python for our app. This problem was solved by using a scheduler called Kala. Which is a simple JSON over HTTP API inpired by Chronus, that does the same as Kala but much more robust and scalable solution by Airbnb . Its totally language agnostic.

Dynamically Schedule Celery Beat task with Flask

2 Answers2