IOerror when using .start() on imported multiprocessing processor subclass

Question

I'm having a problem with python 2.7 multiprocessing (64 bit windows). Suppose I have a file pathfinder.py with the code:

import multiprocessing as mp

class MWE(mp.Process):
    def __init__(self, n):
        mp.Process.__init__(self)
        self.daemon = True
        self.list = []
        for i in range(n):
            self.list.append(i)

    def run(self):
        print "I'm running!"

if __name__=='__main__':
    n = 10000000
    mwe = MWE(n)
    mwe.start()

This code executes fine for arbitrarily large values of n. However if I then import and run a class instance in another file

from pathfinder import MWE

mwe = MWE(10000)
mwe.start()

I get the following traceback if n >= ~ 10000:

Traceback (most recent call last):
  File <filepath>, in <module>
    mwe.start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 280, in __init__
    to_child.close()
IOError: [Errno 22] Invalid argument

I thought this might be some sort of race condition bug, but using time.sleep to delay mwe.start() doesn't appear to affect this behavior. Does anyone know why this is happening, or how to get around it?

score 1 · Accepted Answer · edited May 23 '17 at 11:58

1

The problem is with how you use multiprocessing in Windows. When importing a module that defines a Process class, e.g.:

from pathfinder import MWE

you must encapsulate the running code inside if __name__ == '__main__': block. Therefore, change your client code to read:

from pathfinder import MWE
if __name__ == '__main__':
    mwe = MWE(10000)
    mwe.start()
    mwe.join()

(Also, note that you want to join() your process at some point.)

Check out the Windows-specific Python restrictions doc https://docs.python.org/2/library/multiprocessing.html#windows.

See https://stackoverflow.com/a/16642099/1510289 and https://stackoverflow.com/a/20222706/1510289 for similar questions.

edited May 23 '17 at 11:58

Community

1
1

answered Jan 02 '16 at 17:18

Velimir Mlaker

10,664
4
46
58

Thanks. Unfortunately in the use case where I found this problem, it's caused by importing OTHER files that execute code when imported (setting up singletons and so on); the only calls to Process are in the name =='main' block but apparently the issue still happens. Guess I will have to rewrite around that. – user27118 Jan 02 '16 at 18:40

IOerror when using .start() on imported multiprocessing processor subclass

1 Answers1