Can't use multiprocessing in python function

Question

I'm a beginner of python, I'm trying to put multiprocessing into a function, however python gives me an error.

Please refer the original code as below:

from multiprocessing import Process
import time
def func1():
    print('test1')
    time.sleep(10)  
def func2():
    print('test2')
    time.sleep(5)
if __name__ == '__main__':
    p_func1 = Process(target=func1)
    p_func2 = Process(target=func2)
    p_func1.start()
    p_func2.start()
    p_func1.join()
    p_func2.join()  
    print('done')

It runs well and give the correct result I need.

However, when I tried to put the multiprocessing code into function:

from multiprocessing import Process
import time

def test_multiprocessing():
    def func1():
        print('test1')
        time.sleep(10)  
    def func2():
        print('test2')
        time.sleep(5)
    if __name__ == '__main__':
        p_func1 = Process(target=func1)
        p_func2 = Process(target=func2)
        p_func1.start()
        p_func2.start()
        p_func1.join()
        p_func2.join()  
        print('done')

test_multiprocessing()

Below is error I got, may I know how to fix this issue ? The reason I'd like to put multiprocessing into a funciton is because there is an existing code there, and I don't want to do major change of the code to support multiprocessing.

Traceback (most recent call last):
  File "multipleprocessing.py", line 20, in <module>
    test_multiprocessing()
  File "multipleprocessing.py", line 14, in test_multiprocessing
    p_func1.start()
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'test_multiprocessing.<locals>.func1'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\reduction.py", line 87, in steal_handle
    _winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Access is denied

Per tested code on Linux, it works. Does that mean Windows Python can't support multiprocessing in function?

`AttributeError: Can't pickle local object 'test_multiprocessing..func1'` make `func1` and `func2` module-level functions. — Stop harming Monica, May 21 '18 at 11:19

score 3 · Answer 1 · answered May 21 '18 at 09:32

3

Your code is correct. You shouldn't be keeping if __name__ == '__main__': inside the function. Read more about it here why name=="main"

try like below,

from multiprocessing import Process
import time

def test_multiprocessing():
    def func1():
        print('test1')
        time.sleep(10)  
    def func2():
        print('test2')
        time.sleep(5)

    p_func1 = Process(target=func1)
    p_func2 = Process(target=func2)
    p_func1.start()
    p_func2.start()
    p_func1.join()
    p_func2.join()  
    print('done')

test_multiprocessing()

answered May 21 '18 at 09:32

Prakash Palnati

3,231
22
35

Hi, Prakash. Thanks for reply. I tried to remove if __name__ == '__main__':, however still got the same error there . – Knight May 21 '18 at 09:38
copy paste the exact code which I answered. It works fine for me. Also take care of indentation – Prakash Palnati May 21 '18 at 09:39
Tried to copy and paste your code, still got same error. I'm using Python 3.6.5, is there any version dependency ? – Knight May 21 '18 at 09:46
should not be a problem. Looks like some access is being denied. If you are windows, try running the `cmd` as administrator and run your program – Prakash Palnati May 21 '18 at 09:48
I tested your code on Linux, it works. But in Windows, it still has the same problem. So, Windows python can't support multiprocessing in a function? – Knight May 24 '18 at 14:00
exactly.. some system level permissions issue I suppose. Give python executable the required access and it should work. – Prakash Palnati May 25 '18 at 04:39

Chanda Korat · Answer 2 · 2018-05-25T03:53:27.287

a bit correction in @Prakash answer. You need to call function inside from if __name__== "__main__"

Here, explained well !!

from multiprocessing import Process
import time


def func1():
    print('test1')
    time.sleep(10)

def func2():
    print('test2')
    time.sleep(5)

def test_multiprocessing():
    p_func1 = Process(target=func1)
    p_func2 = Process(target=func2)
    p_func1.start()
    p_func2.start()
    p_func1.join()
    p_func2.join()
    print('done')

if __name__== "__main__":
    test_multiprocessing()

Another way is you can bound method to a class because functions are only picklable if they are defined at the top-level of a module. as below:

from multiprocessing import Process
import time

class Foo:

    def func1(self):
        print('test1')
        time.sleep(10)

    def func2(self):
        print('test2')
        time.sleep(5)

    def test_multiprocessing(self):
        p_func1 = Process(target=self.func1)
        p_func2 = Process(target=self.func2)
        p_func1.start()
        p_func2.start()
        p_func1.join()
        p_func2.join()
        print('done')

if __name__== "__main__":
    f=Foo()
    f.test_multiprocessing()

Thanks for reply Chanda, it works on Windows. If I need to put `func1`, `func2` into function `test_multiprocessing`, is that possible? — Knight, May 24 '18 at 14:05
@Knight please see updated answer. Directly putting `func1`,`func2` into function`test_multiprocessing` would give you pickling error. You can use class instead. — Chanda Korat, May 25 '18 at 03:54

Can't use multiprocessing in python function

2 Answers2