2

I have a function returning a tuple of two elements. The function is called with pool starmap to generate a list of tuples which are unpacked to two lists.

def func():
   #...some operations
   return (x,y)

def MP_a_func(func,iterable,proc,chunk):
    pool=multiprocessing.Pool(processes=proc)
    Result=pool.starmap(func,iterable,chunksize=chunk)
    pool.close()
    return Result
##
if __name__ == '__main__':
    results=MP_a_func(func,iterable,proc,chunk)

a,b=zip(*results)

I now wish to use dask delayed API as the following

if __name__ == '__main__':
    results=delayed(MP_a_func(func,iterable,proc,chunk))

is it possible to unpack tuples in the delayed object without using results.compute() ?

Thank your for your help

Reda E.
  • 653
  • 3
  • 16

2 Answers2

3

It is possible for another delayed function to unpack the tuple, in the example below, the delayed value of return_tuple(1) was not computed, but passed as a delayed object:

import dask

@dask.delayed
def return_tuple(x):
    return x+1, x-1

@dask.delayed
def process_first_item(some_tuple):
    return some_tuple[0]+10

result = process_first_item(return_tuple(1))

dask.compute(result)

As per @mdurant's answer, it turns out delayed function/decorator has nout parameter, also see this answer.

SultanOrazbayev
  • 14,900
  • 3
  • 16
  • 46
2

If you know the number of outputs, the delayed function (or decorator) takes an optional nout arguments, and this will split the single delayed into that many delayed outputs. This sounds like exactly what you need.

mdurant
  • 27,272
  • 5
  • 45
  • 74
  • Interesting, I haven't seen this used before, but it turns out it was around for quite a while: https://stackoverflow.com/questions/44602766/unpacking-result-of-delayed-function – SultanOrazbayev Mar 12 '21 at 20:09