10

I want to have a task that will execute every 5 minutes, but it will wait for last execution to finish and then start to count this 5 minutes. (This way I can also be sure that there is only one task running) The easiest way I found is to run django application manage.py shell and run this:

while True:
    result = task.delay()
    result.wait()
    sleep(5)

but for each task that I want to execute this way I have to run it's own shell, is there an easy way to do it? May be some king custom ot django celery scheduler?

Julian Popov
  • 17,401
  • 12
  • 55
  • 81

6 Answers6

18

Wow it's amazing how no one understands this person's question. They are asking not about running tasks periodically, but how to ensure that Celery does not run two instances of the same task simultaneously. I don't think there's a way to do this with Celery directly, but what you can do is have one of the tasks acquire a lock right when it begins, and if it fails, to try again in a few seconds (using retry). The task would release the lock right before it returns; you can make the lock auto-expire after a few minutes if it ever crashes or times out.

For the lock you can probably just use your database or something like Redis.

Cesar
  • 409
  • 6
  • 14
  • +1. The only person to address the unique instance issue! Details on how to implement a lock if you are using a django database can be found here: http://stackoverflow.com/questions/4095940/running-unique-tasks-with-celery – jcdude May 13 '14 at 16:05
16

You may be interested in this simpler method that requires no changes to a celery conf.

@celery.decorators.periodic_task(run_every=datetime.timedelta(minutes=5))
def my_task():
    # Insert fun-stuff here
Conley Owens
  • 8,691
  • 5
  • 30
  • 43
  • 1
    I got an error 'Celery' object has no attribute 'decorators'. Any idea about this? I wrote @celery.decorators.periodic_task(run_every=datetime.timedelta(minutes=5)) above my task. – Always_a_learner Mar 16 '17 at 11:34
  • The newest version of celery doesn't have this decorator. You'll have to just use the instructions here: http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html – Conley Owens Mar 17 '17 at 17:12
13

All you need is specify in celery conf witch task you want to run periodically and with which interval.

Example: Run the tasks.add task every 30 seconds

from datetime import timedelta

CELERYBEAT_SCHEDULE = {
    "runs-every-30-seconds": {
        "task": "tasks.add",
        "schedule": timedelta(seconds=30),
        "args": (16, 16)
     },
}

Remember that you have to run celery in beat mode with the -B option

manage celeryd -B

You can also use the crontab style instead of time interval, checkout this:

http://ask.github.com/celery/userguide/periodic-tasks.html

If you are using django-celery remember that you can also use tha django db as scheduler for periodic tasks, in this way you can easily add trough the django-celery admin panel new periodic tasks. For do that you need to set the celerybeat scheduler in settings.py in this way

CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
Mauro Rocco
  • 4,980
  • 1
  • 26
  • 40
  • 2
    The problem with this is that it will not wait for task to finish, but will just send another task when it's time (every 30 seconds). Or may be I'm wrong? – Julian Popov Mar 19 '11 at 10:58
  • Thank you for advice, but I think I want something else - I want to create a job, send it for execution, and create another job ONLY when the EXECUTION of previous one is finished. I don't want to create jobs until I know that the previous one is finished. I want the task to have synchronous (not asynchronous) behavior – Julian Popov Mar 20 '11 at 10:01
  • The global objective is to run a task for which I can't tell how much time it will take and when it's finished wait for some time and start it again. Also I have to be sure that it will not be executed 2 or more times simultaneously by different worker threads, and also that I don't have to write my own program code to do this. – Julian Popov Mar 21 '11 at 07:24
  • 2
    If you want to make sure a task only starts after the last one finished, use memcached (or django cache) to create a lock on the task type or resource in said task. Its easy and scalable to do. – michael Sep 07 '12 at 00:20
  • @MauroRocco This is not true, at least as of 3.0.12, `celery beat` will most certainly create overlapping tasks. – Alex B Nov 26 '12 at 05:24
4

To expand on @MauroRocco's post, from http://docs.celeryproject.org/en/v2.2.4/userguide/periodic-tasks.html

Using a timedelta for the schedule means the task will be executed 30 seconds after celerybeat starts, and then every 30 seconds after the last run. A crontab like schedule also exists, see the section on Crontab schedules.

So this will indeed achieve the goal you want.

dkuebric
  • 415
  • 1
  • 3
  • 6
  • Sorry, about the question, but what if task takes 20 seconds to complete, will it run in 0:30 (1-st), finish in 0:50 and then start in 1:20 (this is what I really want) – Julian Popov Mar 19 '11 at 19:50
  • 2
    If you want that the task run every 30 seconds independently from the duration than you have to use a crontab schedule, but remember that this task are added to the celery queue and if there is other tasks in execution/in queue your are not sure that you task start at the given time. – Mauro Rocco Mar 19 '11 at 20:14
2

Because of celery.decorators deprecated, you can use periodic_task decorator like that:

from celery.task.base import periodic_task
from django.utils.timezone import timedelta

@periodic_task(run_every=timedelta(seconds=5))
def my_background_process():
    # insert code
Murat Çorlu
  • 8,207
  • 5
  • 53
  • 78
0

Add that task to a separate queue, and then use a separate worker for that queue with the concurrency option set to 1.

josven
  • 399
  • 4
  • 7