Making a Python decorator that turns any @classmethod function into a function that works on a vector of values in parallel

Question

This is a base Python question. I am trying to decorate a @classmethod with my own decorator, my_decorator, which turns any function into a function which accepts a list of arguments to the original function as an argument. Here's an example.

You have a function foo() and you if you wanted to run it, in sequence, on arguments, a, b, and c. You would do

result_a = foo(a)
result_b = foo(b)
result_c = foo(c)

You could just make a function that works on a list of arguments, L=[a, b, c], and then you could do

results = foo_on_list([a, b, c])

I want to make a decorator that does exactly this for any function foo(). What's more is that I want to use the multiprocessing library to be able to execute foo(a) asynchronously of foo(b). I wrote a nice decorator which works for static functions, but I am having trouble generalizing to functions which must either take a cls or self argument. Here is an example.

import multiprocessing as mp

def my_decorator(method):
    def method_in_vector_form(obj, L, *args, **kwargs):
        pool = mp.Pool(2)
        print('Starting decorator')
        results = [pool.apply_async(method, args=(obj, value)) for value in L]
        pool.close()
        pool.join()
        return [r.get() for r in results]
    return method_in_vector_form

class Foo(object):        
    @classmethod
    def boo(cls, i):
        print('Running from Foo\'s boo' + i)

    @classmethod
    @my_decorator
    def bar(cls, chunk_id):
        print('Running from Foo\'s bar')
        cls.boo(chunk_id)
        return chunk_id * 2

class Child(Foo):
    @classmethod
    def boo(cls, i):
        print('Running from Child\'s version of boo' + i)

if __name__ == '__main__':
    print(Foo.bar([1, 2, 3]))
    print(Child.bar([1, 2, 3]))

I expect the following output (may be out of order because of asynchronicity).

Starting decorator
Running from Foo\'s bar
Running from Foo\'s boo 1
Running from Foo\'s bar
Running from Foo\'s boo 2
Running from Foo\'s bar
Running from Foo\'s boo 3
[2, 4, 6]
Starting decorator
Running from Foo\'s bar
Running from Child\'s version of boo 1
Running from Foo\'s bar
Running from Child\'s version of boo 2
Running from Foo\'s bar
Running from Child\'s version of boo 3
[2, 4, 6]

Here is the current output.

Starting decorator
Traceback (most recent call last):
  File "C:\....\dec_example.py", line 36, in <module>
    print(Foo.bar([1, 2, 3]))
  File "C:\....\TextFromPdf\dec_example.py", line 10, in method_in_vector_form
    return [r.get() for r in results]
  File "C:\....\dec_example.py", line 10, in <listcomp>
    return [r.get() for r in results]
  File "C:\....\pool.py", line 599, in get
    raise self._value
  File "C:\....\pool.py", line 383, in _handle_tasks
    put(task)
  File "C:\....\connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "C:\....\reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function Foo.bar at 0x0000000002897158>: attribute lookup bar on __main__ failed

http://stackoverflow.com/questions/21111106/cant-pickle-static-method-multiprocessing-python — engineerC, Dec 11 '15 at 21:31
That question/answer has nothing to do with writing a custom decorator — under_the_sea_salad, Dec 11 '15 at 21:36
Are you sure to expect `Running from Foo\'s bar` to appear only once? Wouldn't it appear before every call to `boo`? — Pynchia, Dec 11 '15 at 21:36
http://stackoverflow.com/questions/9470403/pickling-class-method — pvg, Dec 11 '15 at 22:03
The issue here is not with your decorator but with the serialization (pickling) needed for inter-process communication. — pvg, Dec 11 '15 at 22:04
@pvg, I understand that the other questions linked to me are similar but I still don't understand how to fix my specific problem. — under_the_sea_salad, Dec 15 '15 at 14:47
It would be helpful if you asked this as 3 separate questions on stackoverflow: (1) How can we write a decorator which accepts a list an input and calls the original function on each element in the list? That is, for input function `f`, the decorator returns a function `wf` such that calling `wf([(a1, a2, ...., an), (b1, b2, ...., bm)])` calls `f(a1, a2, ...., an)` and calls `f(b1, b2, ...., bm)`. (2) What is the proper way to apply our own decorator to a method which is already decorated with '@classmethod'? (3) [character limit reached for comment] — Toothpick Anemone, May 24 '18 at 13:15
(3) How can we parallelize a function `g` which passes each element of a list to a function `f`? Specifically, we would like to use the `multiprocessing` library — Toothpick Anemone, May 24 '18 at 13:17

Making a Python decorator that turns any @classmethod function into a function that works on a vector of values in parallel

0 Answers0