0

I'm trying to use multi-processing on windows 10 with Python 3.10.1. If the method has a decoration, then this code fails:

from multiprocessing import Process

def profiler(func):
    def wrap(*args, **kwargs):
        result = func(*args, **kwargs)
        return result

    return wrap

@profiler
def go_around(num):
    print(num)

if __name__ == '__main__':
    p = Process(target=go_around, args=(1,))
    p.start()
    p.join()

I'm getting this error:

File "", line 1, in File "C:\Python\Python310\lib\multiprocessing\spawn.py", line 102, in spawn_main source_process = _winapi.OpenProcess( OSError: [WinError 87] The parameter is incorrect

Note that this used to work on python 2.7. Any ideas why this is happening and how to fix it?

max
  • 9,708
  • 15
  • 89
  • 144
  • I can't reproduce it on python 3.9, but I got completely another exception: `AttributeError: Can't pickle local object 'profiler..wrap'`. – Olvin Roght Jan 15 '22 at 22:46
  • 1
    See [this gist](https://gist.github.com/EdwinChan/3c13d3a746bb3ec5082f) and the comments below. – azelcer Jan 15 '22 at 23:30
  • 1
    It doesn't work for me in either Python 2.7 or 3.8. Although the error messages are slightly different, they both boil down to the inability to pickle a local function (`wrap()` presumably). – martineau Jan 15 '22 at 23:38

1 Answers1

2

python pickling needs to pull the original function in order to re-construct the decorated function, and it's no longer in the namespace (because it's already been decorated.) using the decorator functools.wraps on the wrapped function stores a copy of the original function in the __dict__ of the new function so it can be accessed later. There's some edge cases where this may not work (classes..) but for simple decorators, it should work:

from multiprocessing import Process
import functools

def profiler(func):
    @functools.wraps(func)
    def wrap(*args, **kwargs):
        result = func(*args, **kwargs)
        print("I profiled your function for you :)")
        return result
    return wrap

@profiler
def go_around(num):
    print(num)

if __name__ == '__main__':
    p = Process(target=go_around, args=(1,))
    p.start()
    p.join()

This answer discusses things a bit further..

Aaron
  • 10,133
  • 1
  • 24
  • 40
  • Great, This works and I do not need to change the usage of my python package. thanks – max Jan 16 '22 at 22:54
  • iirc multiprocessing alternatives that use `dill` instead of `pickle` also won't run into this problem. I tend to try and not stray too far away from the standard libraries however mostly just because it's very well known, stable (not bug free though). – Aaron Jan 17 '22 at 05:01