11

I am trying to figure out how to configure a periodic task in celery to be scheduled to run on load regardless of interval.

For example,

beat_schedule = {
    'my-task': {
        'task': 'module.my_task',
        'schedule': 60.0,
    },
}

will wait 60 seconds after the beat is started to run for the first time.

This is problematic for a longer interval, such as an hour, that can do work that is immediately valuable but is not needed "fresh" at shorter intervals.

This question addresses this issue but neither of the answers are satisfactory:

  1. Adding startup lag for the task to be en-queued is both undesirable performance-wise and bad for maintainability since the initial run and schedule are now separated.

  2. Re-implementing the schedule within the task is bad for maintainability.

This seems to me something that should be obvious, so I am quite surprised that that SO question is all I can find on the matter. I am unable to figure this out from the docs and the celery github issues so I wonder if I am missing something obvious.

Edit:

There seems to be more to the story here, because after trying a different task with an hour interval, it ran immediately as the project celery is started.

If I stop and clear the queue with celery purge -A proj -f then start celery again, the task does not run within the heartbeat interval. This would makes sense because the worker handles the messages but beat has its own schedule record celerybeat-schedule which would be unaffected by the purge.

If I delete celerybeat-schedule and restart beat the task still does not run. Starting celery beat with a non-default schedule db location also does not cause the task to run. The next time the task runs is one hour from the time I started the new beat (14:59) not one hour from the first start time of the task (13:47).

There seems to be some state that is not documented well or is unknown that is the basis of this issue. My question can also be stated as: how do you force beat to clear its record of last runs?.

I am also concerned that while running the worker and beat, running celery -A proj inspect scheduled gives - empty - but presumably the task had to be scheduled at some point because it gets run.

Community
  • 1
  • 1
Jmills
  • 2,414
  • 2
  • 19
  • 22
  • @Jmilis Have you got any workaround yet? – M.A.K. Simanto Feb 08 '19 at 11:51
  • @M.A.K.Simanto I never found a proper solution to this particular problem. An issue I opened on the project GH indicates it _may_ be fixed: https://github.com/celery/celery/issues/3854. I don't have a current use for celery so haven't checked. – Jmills Feb 08 '19 at 16:04

0 Answers0