asynchronous subprocess with Popen

Question

Using Windows 7 + python 2.6, I am trying to run a simulation model in parallel. I can launch multiple instances of the executable by double-clicking on them in my file browser. However, asynchronous calls with Popen result in each successive instance interrupting the previous one. For what it's worth, the executable returns text to the console, but I don't need to collect results interactively.

Here's where I am so far:

import multiprocessing, subprocess

def run(c):
    exe = os.path.join("<location>","folder",str(c),"program.exe")
    run = os.path.join("<location>","folder",str(c),"run.dat")
    subprocess.Popen([exe,run],creationflags = subprocess.CREATE_NEW_CONSOLE)

def main():
    pool = multiprocessing.Pool(3)
    for c in range(10):
        pool.apply_async(run,(str(c),))
 pool.close()
 pool.join()

if __name__ == '__main__':
    main()

After scouring SO for a solution, I've learned that using multiprocessing may be redundant, but I need some way to limit the number of cores working.

Have you tried to specify different working directories for each subprocess using `cwd` argument? The `exe` probably writes some files to current directory that interferes with parallel execution. — jfs, Dec 03 '12 at 05:50
you could use `os.path.join` to join path parts instead of direct string manipulation — jfs, Dec 03 '12 at 05:51
Popen returns immediately so your script doesn't limit number of parallel subprocesses. You could use `subprocess.call()` to block until the subprocess finishes. — jfs, Dec 03 '12 at 05:54
[this answer demonstrates various techniques to limit number of concurrent subprocesses](http://stackoverflow.com/a/9874484/4279): it shows `multiprocessing.Pool`, `concurrent.futures`, `threading + Queue` -based solutions — jfs, Dec 03 '12 at 06:01
@J.F.Sebastian Specifying `cwd` does the trick. I missed that important part in your thorough treatment of the subject, linked above. Please post as an answer. — metasequoia, Dec 03 '12 at 06:10
post your solution as an answer. It is explicitly encouraged. Better to post code that was tested. — jfs, Dec 03 '12 at 06:25

score 1 · Accepted Answer · answered Dec 03 '12 at 06:28

Enabled by @J.F. Sebastian's comment regarding the cwd argument.

import multiprocessing, subprocess

def run(c):
    exe = os.path.join("<location>","folder",str(c),"program.exe")
    run = os.path.join("<location>","folder",str(c),"run.dat")
    subprocess.check_call([exe,run],cwd=os.path.join("<location>","folder"),creationflags = subprocess.CREATE_NEW_CONSOLE)

def main():
    pool = multiprocessing.Pool(3)
    for c in range(10):
        pool.apply_async(run,(str(c),))
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()

btw, there is `mutiprocessing.dummy.Pool` that provides the same interface but uses threads instead of processes that might be more appropriate in this case. — jfs, Jan 06 '13 at 17:30

asynchronous subprocess with Popen

1 Answers1