How to dynamically change the schedule of celery beat?

Question

I am using Celery 4.3.0. I am trying to update the schedule of celery beat every 5 seconds based on a schedule in a json file, so that when I’m manually editing, adding or deleting scheduled tasks in that json file, the changes are picked up by the celery beat scheduler without restarting it.

What I tried is creating a task that update this schedule by updating app.conf['CELERYBEAT_SCHEDULE']. The task successfully runs every 5 seconds but celery beat doesn’t update to the new schedule, even though I set beat_max_loop_interval to 1 sec.

tasks.py

from celery import Celery

app = Celery("tasks", backend='redis://', broker='redis://')
app.config_from_object('celeryconfig')

@app.task
def hello_world():
    return "Hello World!"

@app.task
def update_schedule():
    with open("path_to_scheduler.json", "r") as f:
        app.conf['CELERYBEAT_SCHEDULE'] = json.load(f)

celeryconfig.py

beat_max_loop_interval = 1  # Maximum number of seconds beat can sleep between checking the schedule

beat_schedule = {
    "greet-every-10-seconds": {
        "task": "tasks.hello_world",
        "schedule": 10.0
    },
    'update_schedule': {
        'task': 'tasks.update_schedule',
        'schedule': 5.0
    },
}

schedule.json

{
  "greet-every-2-seconds": {
        "task": "tasks.hello_world",
        "schedule": 2.0
    },
  "update_schedule": {
    "task": "tasks.update_schedule",
    "schedule": 5.0
  }
}

Any help would be appreciated. If you know a better way to reload the beat schedule from a file I’m also keen to hear it.