0

Basically, I want to use a script to start other scripts in different directories and do follow up works until all the "child scripts" are finished. I use multiprocessing.Pool to put all the child process together and use wait() to wait until all finished. Here is my script:

import os, shutil, subprocess,sys, multiprocessing

rootdir=os.getcwd()
argument=[]
def Openpy(inputinfo):
    dirpath, filename= inputinfo
    os.chdir(dirpath)
    return subprocess.Popen( [ 'python',  filename ] )

if __name__=="__main__":
    for dirpath, dirname , filenames in os.walk(rootdir):
        for filename in filenames: 
            if filename=='Test.py':
                argument.append( [ dirpath, filename ] )
    print argument
    po=multiprocessing.Pool()
    r=po.map_async(Openpy, argument)
    po.close()
    r.wait()

print 'Child scripts are finished'

The child processes start normally and finish in one minute. However, it seems that those child processes do not return its "finished message" to the parent process. It did not show that "Child scripts are finished". What can I do to change my code so that it can detect the end of the child processes properly?

All help is appreciated!

2013/9/1 9:16 UTC+8

J.F. Sebastian's answer is being accepted. Run successfully and elegant usage of list comprehension makes the whole stuff more readable.

Ken T
  • 2,255
  • 1
  • 23
  • 30

2 Answers2

0

Try swapping the location of r.wait() and po.close()

if __name__=="__main__":
    for dirpath, dirname , filenames in os.walk(rootdir):
        for filename in filenames: 
            if filename=='Test.py':
                argument.append( [ dirpath, filename ] )
    print argument
    po=multiprocessing.Pool()
    r=po.map_async(Openpy, argument)
    r.wait()
    po.close()

It might not be your problem, but it looks like maybe you are closing the pool before the result is evaluated.

user2719952
  • 11
  • 1
0

Popen() returns immediately so your parent process finishes long before your child scripts i.e., look for the "finished message" at the very beginning of the output.

You don't need multiprocessing to run subprocesses concurrently:

import os
import sys
from subprocess import Popen

# run all child scripts in parallel
processes = [Popen([sys.executable, filename], cwd=dirpath)
             for dirpath, dirname , filenames in os.walk('.')
             for filename in filenames
             if filename == 'Test.py']

# wait until they finish
for p in processes:
    p.wait()
print("all done")

See also Python threading multiple bash subprocesses?

Community
  • 1
  • 1
jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • Your method works. But can `Popen` utilize my multi-core cpu? Btw, `from subprocess import Popen` should be added. – Ken T Aug 31 '13 at 07:55
  • @user2720402: yes, `Popen` can utilize multiple cpus (each call creates a separate process). – jfs Aug 31 '13 at 16:29
  • Thanks man! One more question but a little bit unrelated to the topic. Is that only the module `threading.thread` cannot utilize the multiple cores cpu when calling many processes? – Ken T Sep 01 '13 at 01:15
  • @user270402: unless you take special measures, multiple processes can utilize multiple CPUs – jfs Sep 01 '13 at 08:42