Python multiprocessing Process() doesn't work on Windows 10 (even for pickling simple strings)

Question

I wrote a Python class to plot pylots in parallel. It works fine on Linux but when I tried it on Windows I ran into problems. It went even that far that I can't pickle a single string. I always end up getting this error:

Traceback (most recent call last):
  File "test.py", line 50, in <module>
    test()
  File "test.py", line 7, in test
    asyncPlotter.saveLinePlotVec3("test")
  File "test.py", line 41, in saveLinePlotVec3
    args=(test, ))
  File "test.py", line 34, in process
    p.start()
  File "C:\Users\adrian\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\adrian\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\adrian\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\adrian\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\adrian\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle weakref objects

C:\Python\MonteCarloTools>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\adrian\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "C:\Users\adrian\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 82, in steal_handle
    _winapi.PROCESS_DUP_HANDLE, False, source_pid)
OSError: [WinError 87] The parameter is incorrect

I narrowed it down to the spawn start method that Windows uses (https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods), when I use the same start method on Linux I can reproduce the error.

import multiprocessing as mp
def test():

    asyncPlotter = AsyncPlotter()

    asyncPlotter.saveLinePlotVec3("test")

    asyncPlotter.saveLinePlotVec3("test")

    asyncPlotter.saveLinePlotVec3("test")

    asyncPlotter.join()


class AsyncPlotter():

    def __init__(self, processes=mp.cpu_count()):

        self.manager = mp.Manager()
        self.nc = self.manager.Value('i', 0)
        self.pids = []
        self.processes = processes


    def linePlotVec3(self, nc, processes, test):

        self.waitOnPool(nc, processes)

        return None


    def waitOnPool(self, nc, processes):

        while nc.value >= processes:
            time.sleep(0.1)
        nc.value += 1


    def process(self, target, args):

        p = mp.Process(target=target, args=args)
        p.start()
        self.pids.append(p)


    def saveLinePlotVec3(self, test):

        self.process(target=self.linePlotVec3,
                       args=(self.nc, self.processes, test))


    def join(self):
        for p in self.pids:
            p.join()


if __name__=='__main__':
    test()

I expect that I can pickle any kind of string, list, dict etc with this method on Windows as well. I'd appreciate if someone could find a way to make this work on Windows. Thanks!

Possible duplicate of [OSError: \[WinError 87\]The parameter is incorrect](https://stackoverflow.com/questions/48627793/oserror-winerror-87the-parameter-is-incorrect) — SF12 Study, Jul 24 '19 at 06:16

Mars · Answer 1 · 2019-07-24T07:14:44.070

2

Change

self.process(target=self.linePlotVec3, args=(test, ))

to

self.process(target=AsyncPlotter.linePlotVec3, args=(test, ))

Process takes in class or global methods, so it's crashing when you try to pass it an instance method

edited Jul 24 '19 at 07:14

answered Jul 24 '19 at 06:31

Mars

2,505
17
26

1

thanks for the fast response! I am getting there, but I forgot to include a part in my example that now gives me another error. When I change to ```self.process(target=AsyncPlotter.linePlotVec3, args=(test, ))``` the self instance in linePlotVec3 is not recognized anymore I think: ```TypeError: linePlotVec3() missing 1 required positional argument: 'test'``` – Adrian Jul 24 '19 at 06:59
@Adrian What's the error now? It runs just fine for me. – Mars Jul 24 '19 at 07:00
I changed the code in my original post to reflect my new error. – Adrian Jul 24 '19 at 07:06
I'm not exactly sure what you mean with that, sorry! can you give me an example? Also why is this not needed on Linux? – Adrian Jul 24 '19 at 07:10
it doesn't work, I am getting the same error again :( – Adrian Jul 24 '19 at 07:13
@Adrian Sorry, my answer is still correct, but it won't help with your new code/ what you're trying to achieve. If you want to pass in instances, check this out: https://stackoverflow.com/a/27320254/3838167 – Mars Jul 24 '19 at 07:30
from the answer in your link: "Also note that in Python 3.x, pickle can pickle instance method types natively, so none of this stuff matters any more"... I am using Python 3.7.3 so it should be working natively as far as I understand? – Adrian Jul 24 '19 at 07:34
@Adrian Apparently not, at least for windows. – Mars Jul 24 '19 at 07:45
I tried it with Python 3.6.3 before already and had the same issue :( – Adrian Jul 24 '19 at 13:52
@Adrian Interesting. Then all I can suggest is converting your logic to use static methods. Since it looks like you're only using one AsyncPlotter, that shouldn't be an issue – Mars Jul 25 '19 at 00:19

Python multiprocessing Process() doesn't work on Windows 10 (even for pickling simple strings)

1 Answers1