65

I have a task in Celery that could potentially run for 10,000 seconds while operating normally. However all the rest of my tasks should be done in less than one second. How can I set a time limit for the intentionally long running task without changing the time limit on the short running tasks?

jimstandard
  • 1,067
  • 2
  • 10
  • 17

2 Answers2

89

You can set task time limits (hard and/or soft) either while defining a task or while calling.

from celery.exceptions import SoftTimeLimitExceeded

@celery.task(time_limit=20)
def mytask():
    try:
        return do_work()
    except SoftTimeLimitExceeded:
        cleanup_in_a_hurry()

or 

mytask.apply_async(args=[], kwargs={}, time_limit=30, soft_time_limit=10)
Ikar Pohorský
  • 4,617
  • 6
  • 39
  • 56
mher
  • 10,508
  • 2
  • 35
  • 27
  • note that the timeout/soft_timeout arguments to apply_async is only available in the development version of celery (master branch, the future version 3.1) – asksol Jul 27 '12 at 14:32
  • 13
    You can also set time-limits for a task in configuration by using: `CELERY_ANNOTATIONS = {'module.mytask': {'time_limit': 20.0}}` – asksol Jul 27 '12 at 14:33
  • 8
    For celery versions 3.1.x it looks like the `timeout/soft_timeout` arguments to `apply_async` have also been changed to `time_limit/soft_time_limit`. The relevant change was introduced here - https://github.com/celery/celery/commit/be6cef2e441e5ecf5857aeb77bd885f06128b9c9 – sanchitarora Dec 14 '15 at 16:11
  • adding to @sanchitarora comment, a link from 3.1.20 docs - http://docs.celeryproject.org/en/latest/userguide/tasks.html#Task.soft_time_limit – ErezO Feb 07 '16 at 06:18
14

This is an example with decorator for an specific Task and Celery 3.1.23 using soft_time_limit=10000

@task(bind=True, default_retry_delay=30, max_retries=3, soft_time_limit=10000)
def process_task(self, task_instance):
   """Task processing."""
        pass
gogasca
  • 9,283
  • 6
  • 80
  • 125