Monitor spawned child process to complete before spawning another child process

Question

I need some directions for the following issue.

I have 6 processes that I need to run. These are PHP files that I need to spawn from shell script. I am able to do it but the results are not what I want.

The sequence of execution is given below.

Process1 needs to be submitted and unless it completes process 2 to process 5 cannot be submitted.
Process 2 to process 5 can be submitted in parallel.
Process final needs to be submitted after process 2 to process 5 are completed.

I have searched various pages but am unable to find a resolution to it. The pages I have referenced are given below.

https://unix.stackexchange.com/questions/76717/bash-launch-background-process-and-check-when-it-ends

Waiting for background processes to finish before exiting script

Whatever I did following the above links, all the process gets submitted at the same time. Please guide me. The code is given below.

#!/bin/sh
start=`date +%s`
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process1 
wait
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process2
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process3
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process4
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process5

wait
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=final

end=`date +%s`
runtime=$((end-start))
echo "$runtime"

Please be cognizant that I am newbie and do not know much about the shell process.

Will appreciate any directions and feedback.

Thanks

score 1 · Accepted Answer · answered Nov 19 '17 at 10:20

1

You can run a program in the background by adding & at the end, and you can use wait to wait for backgrounded processes to complete:

#!/bin/sh

start=$(date +%s)

php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process1 

php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process2 &
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process3 &
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process4 &
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process5 &

wait

php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=final

end=$(date +%s)
echo $((end - start))

Btw, you should avoid copy-paste coding. Put repeated values in a variable so you can make changes later in one place, and extract common logic to helper functions.

#!/bin/sh

run() {
    local process_name=$1
    php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name="$process_name"
}

start=$(date +%s)

run process1 

for i in 2 3 4 5; do
    run process$i &
done

wait

run final

end=$(date +%s)
echo $((end - start))

answered Nov 19 '17 at 10:20

janos

120,954
29
226
236

Tried that also, but everything gets submitted at the same time and does not as I wanted. – user7859160 Nov 19 '17 at 10:25
@user7859160 what I wrote submits process2, 3, 4, 5 in parallel, but not 1 and final. So I don't understand your comment. – janos Nov 19 '17 at 10:26
Process 2-5 needs to wait till process 1 is completed. Also process final needs to wait till process 1-5 is completed. When I submit your modified code or my code, it submits everything at once and the script run time it gives is 0 seconds. If I run these process individually it takes around 25 seconds. – user7859160 Nov 19 '17 at 11:16
The answer from @janos looks good to me. Does the PHP code spawn processes? I see that you are using `sh`. @user7859160, maybe you could try with `bash`? – Javier Elices Nov 19 '17 at 14:48
@janos: Thanks for the solution. It worked flawlessly. Also thanks to Javier's comments which made me look at my PHP code again. There was an issue with my PHP code in accepting arguments. – user7859160 Nov 19 '17 at 18:26

score 0 · Answer 2 · answered Nov 19 '17 at 18:32

Solution provided by @janos was great, but the issue was with the way I was calling the PHP code. Calling PHP script from browser is different than from Shell. Modified code to accept both values from browser and from shell is given below.

if ($argc <2) //positional argument count passed from shell
{//Execute this when the browser makes the call
    if ((isset($_REQUEST['user_id'])) && (!empty($_REQUEST['user_id'])))
    {
            $user_id = $_REQUEST['user_id'];
        echo 'Hi';
        }else{

            $output['error'] = "Invalid User.";
        }
    if ((isset($_REQUEST['process_name'])) && (!empty($_REQUEST['process_name'])))
    {
        $process_name = $_REQUEST['process_name'];

    }else{//Execute this when call is made from shell

        $output['error'] = "Invalid Process Name";
    }

}
else
{
    $user_id = $argv[1]; //positional argument value passed from shell
    $process_name = $argv[2]; //positional argument value passed from shell
}

score 0 · Answer 3 · answered Dec 20 '17 at 00:26

With GNU Parallel it looks like this:

#!/bin/sh
start=`date +%s`
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=process1 
parallel php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name={} ::: process{2..5}
php /usr/share/nginx/html/cron/complete-api-fetch.php user_id=6 process_name=final

end=`date +%s`
runtime=$((end-start))
echo "$runtime"

Monitor spawned child process to complete before spawning another child process

3 Answers3