Python multiprocessing.Pool is weird on Windows

Question

I have a super simple python script as defined here

import multiprocessing
from multiprocessing import Pool

print "CPUs: " + str(multiprocessing.cpu_count())
convertPool = Pool(multiprocessing.cpu_count())

On Linux this appears to behave as I would of expected, with the program just starting and then printing out the number of cores, then exiting. However, on Windows this program will continue to invoke new python commands and print out "CPUs: 4" (I have 4 cores) and never stop running and will eventually kill the box. Can someone please explain what is going on here?

Thanks

EDIT: I now have the following program which still doesn't operate as I would expect

import sys
import os
import subprocess
from subprocess import Popen, PIPE
from threading import Thread
import multiprocessing
from multiprocessing import Pool, freeze_support

try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # python 3.x

ON_POSIX = "posix" in sys.builtin_module_names

def myPrint(line, logWriter):
    if logWriter is None:
        # This will print without a newline, cause the process
        # has its own newlines
        sys.stdout.write(line)
    else:
        logWriter.write(line.strip())
        logWriter.write("\n")
        logWriter.flush()

# This is gotten from http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python
def executeCommand(cmd, logWriter):
   myPrint(cmd + "\n", logWriter)
    p = Popen(cmd, stdout=PIPE, bufsize=4096, close_fds=ON_POSIX, shell=True)
    q = Queue()
    t = Thread(target=enqueue_output, args=(p.stdout, q))
    t.daemon = True # thread dies with the program
    t.start()

    # read line without blocking
    while t.isAlive() or not q.empty():
        try:
            line = q.get_nowait() # or q.get(timeout=.1)
        except Empty:
            pass # Do nothing
        else: # If there is a line, then print it 
            if logWriter is not None:
                myPrint(line, logWriter)
        # Sleep to prevent python from using too much cpu
        time.sleep(0.05)

if __name__ == "__main__":
    freeze_support()
    convertPool = Pool(multiprocessing.cpu_count())
    # Now let's smooth and threshold all the masks
    for foo in bar:
        threshSmoothCmd = "ConvertImage.exe -i " + foo + " -o " + foo
        myPrint(threshSmoothCmd + "\n", None)
        convertPool.apply_async(executeCommand,(threshSmoothCmd, None,))

    convertPool.close()
    convertPool.join()
print "Finished processing"

ConvertImage.exe is an executable I wrote myself foo and bar are just placeholders. On Linux this will fire up multiprocessing.cpu_count() number of ConvertImage.exe processes and then print Finished processing once all of the ConvertImage.exe processes have finished. On Windows this immediately fires up len(bar) ConvertImage.exe processes and then immediately prints Finished processing and exits. How do I get the Windows version to behave like the Linux version?

Include `if __name__ == "__main__":` to make this work. this is covered in the `multiprocessing` documentation. — Sven Marnach, Jul 16 '12 at 17:09
Also note that, in my opinion, it's not `multiprocessing.Pool` that is weird. The weird thing is Windows lacking `fork()`. — Sven Marnach, Jul 16 '12 at 17:10
possible duplicate of [python3.x multiprocessing cycling without "if __name__ == '__main__':"](http://stackoverflow.com/questions/11501048/python3-x-multiprocessing-cycling-without-if-name-main) — Sven Marnach, Jul 16 '12 at 17:13
@SvenMarnach -- Thanks for your quick response. I have updated my question accordingly. — Jon, Jul 16 '12 at 17:27
If you want to ask a new question, please ask a new question and do not edit an old one. StackOverflow is meant to be a collection of questions that is useful for later reference. Please read the Windows programming guidelines in the `multiprocessing` documentation before posting a new question. — Sven Marnach, Jul 16 '12 at 17:29
Is this the section you are referring to? http://docs.python.org/library/multiprocessing.html#windows — Jon, Jul 16 '12 at 17:32
Yes, exactly. There are a few issues to be aware of on Windows. — Sven Marnach, Jul 16 '12 at 17:41
So even if I add the freeze_support() line right after if __name__ == "__main__": I still get the same response. What else am I missing? — Jon, Jul 16 '12 at 17:50
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/13959/discussion-between-jon-and-sven-marnach) — Jon, Jul 16 '12 at 17:54

score 0 · Accepted Answer · answered Jul 16 '12 at 18:07

0

I figure this out, it was working correctly, my example program was posted wrong as I forgot some imports and that caused the described behavior. After I got all the imports correct this example worked fine on Windows and Linux.

answered Jul 16 '12 at 18:07

Jon

3,985
7
48
80

Python multiprocessing.Pool is weird on Windows

1 Answers1