1

I'm trying to schedule periodic tasks at runtime in Celery.

Here is an older question that is relevant. One of the answers in that question mentions that it is now possible to add tasks during runtime since Celery version 4.1.0 (I'm using 4.3.0) but I am not able to achieve that.

I'm modifying the shelve database while the Celery beat is running (or even when it's not running). Here's the code doing that:

import shelve
from celery.beat import ScheduleEntry
from celery.schedules import schedule

s = schedule(5, app=app)
entry = ScheduleEntry("Test Task", "proj.tasks.my_task", args=("arg1", "arg2"), schedule=s)
db = shelve.open("celerybeat-schedule", writeback=True)
db["entries"]["Test Task"] = entry
db.close()

This is stored in the shelve file successfully. Here's the entries field that shelve file:

{'Test Task': <ScheduleEntry: Test Task proj.tasks.my_task('arg1', 'arg2') <freq: 5.00 seconds>}

But as soon as the celery beat starts, it over-writes this file and makes entries field an empty dict. Am I missing something? Or this is not possible without using Django or similar scheduler?

Nijraj Gelani
  • 1,446
  • 18
  • 29

1 Answers1

0

As it turns out, this is still not possible. I switched over to using celerybeat-mongo which conveniently stores all the schedules in a Mongo collection and also loads from the same collection that might have been inserted by some other process.

Nijraj Gelani
  • 1,446
  • 18
  • 29