Detecting termination of multiple child processes with multiprocessing

Question

#!/usr/bin/python
import os

from os import getpid

import multiprocessing


build="613719"
file1=open('/auto/home/venkam11/python/install-script/build-ddr-file.txt', 'r')

def installation(model,ddr,build):

    cli = "/auto/tools/qa/shared/qa-branch/util/install.pl -durham -restart -silentinstall -model %s -branch 6.2A %s %s"  %(model, ddr, build)

    print cli
    os.popen2(cli)
    print "installation has started on %s \n" %ddr

if name == 'main':

pid=getpid()

print("parent process id :{}".format(getpid()))

for ddr in file1:
    print ddr.rstrip()
    if 'dd4500' in ddr:
        print "dd4500"
        model = "dd4500"
    elif ('apollo' or 'apolloplus') in ddr:
        print "dd9500"
        model = "dd9500"
    elif 'dd2500' in ddr:
        print "dd2500"
        model = "dd2500"
    elif 'dd7200' in ddr:
        print "dd7200"
        model = "dd7200"
    elif 'jupiter' in ddr:
        print "dd9800"
        model = "dd9800"
    ddr = ddr.rstrip()
    ins=multiprocessing.Process(target=installation, args=(model,ddr,build))
    ins.start()

Basically iam trying to read the file which has the machine names and using multiprocessing, I want to insatll the OS on the machines which I have read.

Above is my code, when I run, it starts installing on all the machines at once and the main program terminates.

But I want the main program not to terminate, it has to wait until the child process finish the job, also return the output saying the child process job is complete.

Install make take anytime 1 hr or 2 hrs, but I want the message saying that all the process jobs are completed.

Can anyone please help here.

Answer 1

Populate a list of processes and use join() to join them with the parent process and then print the message. This way, the parent waits till the children are done with their tasks before executing the lines that follow.

The code for the same should look something like this:

#!/usr/bin/python
import os
import multiprocessing
import subprocess
import time
import sys

from os import getpid

file1 = open('/auto/home/venkam11/python/install-script/build-ddr-file.txt', 'w+')
print ("enter the ddr names one by one in each line and press ctrl-d twice")
userInput = sys.stdin.readlines()
file1.writelines(userInput)
file1.close()
build = input("\nenter the build number : \n")
branch = raw_input("enter the branch name  : " )
file1 = open('/auto/home/venkam11/python/install-script/build-ddr-file.txt', 'r')

def installation(model, branch, ddr, build, shared_dict):
        cli = "/auto/tools/qa/shared/qa-branch/util/install.pl -durham -restart -silentinstall -model %s -branch %s %s %s"  %(model, branch, ddr, build)
        print cli
        print "installation has started on %s \n" % ddr
        time.sleep(20)
        try:
            subprocess.check_call(cli, shell=True)
            shared_dict[ddr] = True
        except subprocess.CalledProcessError:
            shared_dict[ddr] = False

if __name__ == '__main__':
    #pid=getpid()
    #print("parent process id : {}".format(getpid()))
    processes = []
    manager = multiprocessing.Manager()
    shared_dict = manager.dict()
    for ddr in file1:
        print ddr.rstrip()
        if 'dd4500' in ddr:
            print "dd4500"
            model = "dd4500"
        elif ('apollo' or 'apolloplus') in ddr:
            print "dd9500"
            model = "dd9500"
        elif 'dd2500' in ddr:
            print "dd2500"
            model = "dd2500"
        elif 'dd7200' in ddr:
            print "dd7200"
            model = "dd7200"
        elif 'jupiter' in ddr:
            print "dd9800"
            model = "dd9800"

        ddr = ddr.rstrip()
        ins = multiprocessing.Process(target=installation, args=(model, branch, ddr, build, shared_dict))
        ins.start()
        processes.append(ins)

    for process in processes:
        process.join()

    print('All the installations are complete')
    print('Details: ')
    for ddr, success in shared_dict.items():
        if success:
            print('Installation on {} successful'.format(ddr))
        else:
            print('Installation on {} unsuccessful'.format(ddr))

Answer 2

Welcome to Stackoverflow. Much like threads, the easiest way to synchronise with your subprocesses is to join() them, most usually in the creating thread/process. The os.popen2 call was deprecated when the subprocess module was introduced, as it does not give the necessary degree of control over and communication with the started subprocess. For that you have correctly deduced you can use the multiprocessing module.

The technical answer to your question using multiprocessing is already covered well in another answer, as well as in this question, which explains how the calling entity can synchronise with its child processes. It's quite permissible to deal with an unknown number of subprocesses by storing each one in a list, for example, like this

for p in my_processes:
    p.join()

A more useful answer might be that the subprocess module allows creation and monitoring of other processes without the overhead and complexity of multiprocessing, which has advanced features for inter-process communication that you do not (yet) appear to need. Its documentation has quite good examples of how older and/or less appropriate Python code can be adapted to use the new module, so it could be worth a look.