4

How do I chunk tasks with keyword arguments? For example, this task:

@app.task
def add(x, y, multiply=1, unit="GB"):
    return '%s %s' % ((x + y) * multiply, unit)

I can call this task as usual with add.apply_async(args=(1, 2), kwargs={'unit': 'MB'}) but how do I chunk it?

I want to do something like : 

add.chunks([{'args': (1, 2), 'kwargs': {'unit': 'MB'}}, {'args': (3, 4), 'kwargs': {'unit': 'KB'}}, ...], 10)

The documentation only shows how to use chunks with positional arguments:

from proj.tasks import add
res = add.chunks(zip(range(100), range(100)), 10)()
arthur
  • 2,319
  • 1
  • 17
  • 24
Oskar Persson
  • 6,605
  • 15
  • 63
  • 124

1 Answers1

2

From what I see in chunks source code you have to give an iterable of tuple of arguments.

So you can make this kind of helper : 

def build_kwargs_tuples(params, default_multiply=1, default_unit="GB"):
    for d in params:
        x,y = d["args"]
        kwargs = d.get("kwargs",{})
        multiply = kwargs.get("multiply", default_multiply)
        unit = kwargs.get("unit", default_unit)
        yield (x,y,multiply, unit)

Then you can specify your arguments the way you ask : 

In [40]: example = [{'args': (1, 2), "kwargs": {'unit': 'MB'}}, {'args': (3, 4), "kwargs": {'unit': 'KB'}}]

In [41]: res = add.chunks(build_kwargs_tuples(example),1)()

In [43]: res.get()
Out[43]: [[u'3 MB'], [u'7 KB']]
arthur
  • 2,319
  • 1
  • 17
  • 24
  • This only uses positional arguments, not keyword arguments – Oskar Persson Feb 16 '17 at 22:06
  • In the definition of my task, there are only keyword arguments (like in yours). It's only when I call the task that I use their position. Maybe I did not understand what you wanted to achieve, in this case please post a piece of code and the desired behavior – arthur Feb 17 '17 at 09:52
  • How you would like to chunk the tasks ? how your arguments should vary ? I mean do you want to apply this task with `x in range(100)` `y in range(100)` and fixed `unit = "GB" `? – arthur Feb 17 '17 at 10:15
  • I want to do something like `add.chunks([{'args': (1, 2), kwargs: {'unit': 'MB'}}, {'args': (3, 4), kwargs: {'unit': 'KB'}}, ...], 10)` – Oskar Persson Feb 17 '17 at 10:21