Celery timeout in Django

Question

There are altogether eight tasks running in celery in different periods. All of them are event-driven tasks. After a certain event, they got fired. And the particular task works continuously until certain conditions were satisfied.

I have registered a task which checks for certain conditions for almost two minutes. This task works fine most of the time. But sometimes the expected behavior of the task is not attained.

The signature of the task is as below:

tasks.py

import time
from celery import shared_task

@shared_task()
def some_celery_task(a, b):
    main_time_end = time.time() + 120
    while time.time() < main_time_end:
        ...
        # some db operations here with given function arguments 'a' and 'b' 
        # this part of the task get execute most of the time

    if time.time() > main_time_end:
        ...
        # some db operations here.
        # this part is the part of the task that doesn't get executed sometimes

views.py

# the other part of the view not mentioned here
# only the task invoked part 
some_celery_task.apply_async(args=(5, 9), countdown=0)

I am confused about the celery task timeout scenarios. Does that mean the task will stop from where it timeouts or will retry automatically? It will be a great help if any clear idea about timeout and retries you guys got.
What could be the reason behind the explained scenarios above? Any help on this question will be highly appreciated. Thank you.

How do you know it's timing out? You should add some logging inside your task so you can check your logs and understand better what's going on. Note that if `time.time() == main_time_end` your task just exits. — dirkgroten, Jan 13 '20 at 11:25
I did not know whether it was timing out or not. Because this task is running for a longer period and the timing out word started buzzing in my head. I think this ```python time.time() == main_time_end``` is the part I was missing. Thank you very much @dirkgroten. — dipesh, Jan 13 '20 at 11:40
Hope it could help https://stackoverflow.com/questions/11672179/setting-time-limit-on-specific-task-with-celery — Sergey Pugach, Jan 13 '20 at 11:58

score 6 · Accepted Answer · answered Jan 13 '20 at 22:09

Check Celery documentation on Tasks - basics are documented very well.

If task fails or was terminated - task will have states.FAILURE status. It will not be re-tried unless specifically coded. If logging is correctly configured - you might see exception messages in logs in case of timeouts or other code exceptions.

When Celery Task TIME_LIMIT is exceeded - task is terminated right away:

The worker processing the task will be killed and replaced with a new one.

Also, TimeLimitExceeded exception will be raised with message like Task handler raised error: "TimeLimitExceeded(2700)"

If Celery SOFT_TIME_LIMIT is set and is smaller than TIME_LIMIT and is exceeded - than SoftTimeLimitExceeded exception will be raised allowing it to be catched in the task and perform clean-up actions.

When worker consumes message (task) from the broker queue - broker needs to know that the message was consumed successfully. To confirm successful consumpion of message worker acknowledges (ACK) to broker. Until message is not acknowledged it is not deleted from broker but also not available for consumption ("invisible"). In not acknowledged - message will be re-delivered back to broker queue available again for consumption.

Redelivering un-acknowledged messages logic depends on broker:

AMQP (RabbitMQ) broker - tracks connection status with worker, and if connection is lost - returns message back to queue.
Redis or SQS broker has its own timeout after which message will be re-delivered to broker queue if not ACKed.

By default celery worker acknowledges message right at the start of the task.

If ACKS_LATE is set - worker acknowledges to broker only after successfully executing task.

One can RETRY task, by catching exception in the task and sending same task back to the broker for re-execution - then this same task with same id will be queued at broker. Countdown option allows to specify delay before the task will be retried.

Celery Task Execution and other Options can be set globally in settings.py or per task as arguments.

Recommended way it to design tasks / logic with consideration of such events to be totally legit and see them normal (but not actually expected) to happen sometime and be ready:

tasks may fail (next same task may do work for both or checks that specific work was not done and re-fire task)
same task may run again (idempotency)
similar tasks can be run simultaneously (locking)

Celery timeout in Django

1 Answers1