Celery task best practices in Django/Python

Question

Consider a function that requires some heavy lifting to be done asynchronously. Calling clients can either recive a cached version immediately, or recive a response that the numbers are being crunched (a valid response for the clients).

Is the following snippet a sound implementation for this pattern?

from django.core import cache
from proj.celery import app

class SomeModel(models.Model):
    # ...
    def get_crunched_numbers(self):
        cache_key = 'foo:{}'.format(self.id)
        res = cache.get(cache_key)
        if not res:
            @app.task
            def gen_crunched_numbers():
                res = do_heavy_lifting()
                cache.set(cache_key, res)
                return res
            gen_crunched_numbers.delay()
            return 'crunching... come back later'
        else:
            return res

Are there better alternatives to running Celery tasks like so, while containing all the logic in a single piece of code?

Edit: as mentioned in the comments, this code doesn't even work. So any suggestions on a nice pattern are much obliged.

Does that even work at all? I would be surprised if a nested function like that could be serialized. Much better surely to make `cache_key` a parameter to a completely standalone function. — Daniel Roseman, Aug 03 '14 at 14:09
@DanielRoseman you are correct this only worked locally (when using `ALWAYS_EAGER`=True) and is totally unserializable. — Yuval Adam, Aug 03 '14 at 15:45

score 2 · Accepted Answer · answered Aug 03 '14 at 20:30

Your code looks very confusing. Why don't you define celery task function outside of your class and call it like this:

from django.core import cache
from proj.celery import app

class SomeModel(models.Model):
    # ...
    def get_crunched_numbers(self):
        cache_key = 'foo:{}'.format(self.id)
        res = cache.get(cache_key)
        if not res:
            gen_crunched_numbers.delay(cache_key)
            return 'crunching... come back later'
        else:
            return res

@app.task
def gen_crunched_numbers(cache_key):
    res = do_heavy_lifting()
    cache.set(cache_key, res)
    return res

Also I usually prefer to create tasks with bind=True:

@app.task(bind=True)
def gen_crunched_numbers(self, cache_key):
    res = do_heavy_lifting()
    cache.set(cache_key, res)
    return res

Which give me access to task context through self.request. For example to change behavior depending if function is called through celery or directly:

@app.task(bind=True)
def gen_crunched_numbers(self, cache_key):
    res = do_heavy_lifting()
    cache.set(cache_key, res)
    if self.request.called_directly:
        return res
    else:
        return { 'result': res, 'cache': cache_key }

I was trying to encapsulate all the logic in a single function, but as you saw, it doesn't really work. Your suggestion looks good, and it's what I'm currently using. The `bind=True` is a very good improvement and I'll likely use it as well. Thanks! — Yuval Adam, Aug 04 '14 at 09:48

Celery task best practices in Django/Python

1 Answers1

Linked