Parallel Decorator with Apply_async()

Question

I would like to use multiprocessing in Python in order to process a high computational cost function in k times and gather returned results in a list. Let me show my function like

def _heavy_func(value):
        a, b = 0, 1
        for item in range(value):
                a, b = b, a + b
        import time
        time.sleep(1.3)
        return a

Then, I invoke the _heavy_func to show non-parallel manner

In [1]: print [ _square_and_offset(i) for i in range(12)]

Its performance is

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
IPython CPU timings (estimated):
  User   :      13.01 s.
  System :       0.00 s.
Wall time:      13.01 s.

Now I have accommodated decorator-oriented parallelism to my function like

from functools import wraps
from multiprocessing import Pool

def parallel_decor(n_procs=None):
    def _parallel_decor(function):
        @wraps(function)
        def __parallel_decor(iterable_values, *args, **kwargs):
            args = list(args)
            p = Pool(n_procs)
            result = [p.apply_async(function, args=[value]+args, kwds=kwargs) for value in iterable_values]
            p.close()
            try:
                return [r.get() for r in result]
            except KeyError:
                return []
        return __parallel_decor
    return _parallel_decor

@parallel_decor(5) # 5 processes
def _heavy_func(value):
        a, b = 0, 1
        for item in range(value):
                a, b = b, a + b
        import time
        time.sleep(1.3)
        return a

def main():
    _heavy_func(range(12))

if __name__ == "__main__":
    main()

Unfortunately, I got an error

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
C:\Users\Laptop\Desktop\sp_parallel.py in <module>()
     30 
     31 if __name__ == "__main__":
---> 32         main()

C:\Users\Laptop\Desktop\sp_parallel.py in main()
     27 
     28 def main():
---> 29         _heavy_func(range(12))
     30 
     31 if __name__ == "__main__":

C:\Users\Laptop\Desktop\sp_parallel.py in __parallel_decor(iterable_values, *args, **kwargs)
     11                         p.close()
     12                         try:
---> 13                                 return [r.get() for r in result]
     14                         except KeyError:
     15                                 return []

C:\Anaconda\lib\multiprocessing\pool.pyc in get(self, timeout)
    565             return self._value
    566         else:
--> 567             raise self._value
    568 
    569     def _set(self, i, obj):

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

According to PicklingError in multiprocessing, the following types can be pickled is functions defined at the top level of a module. My question is if there is any chances to modify my approach to have pickled function with-in a decorator?

Thanks in advance

Parallel Decorator with Apply_async()

0 Answers0