how to control the timing of call to scripts within a shell wrapper

Question

OS: fedora fc14.x86_64:

I want to create a wrapper that executes multiple scripts, and, have each successive script not begin until all the jobs in the prior script have completed.

In the below scripts, 'execute_swirl' jobs need 'vanilla' and 'chocolate' being finished before executing.

What is happening is that the successive scripts (for example, script 'chocolate' below) is starting before 'vanilla' is complete.

# execute all the jobs related to vanilla
execute_vanilla.sh;

execute_vanilla.sh contains:
/usr/local/bin/program/job1 & /usr/local/bin/program/job2 & /usr/local/bin/program/job3;


sleep 60;

# execute all the jobs related to chocolate
execute_chocolate.sh;

execute_chocolate.sh contains:
/usr/local/bin/program/job4 & /usr/local/bin/program/job5 & /usr/local/bin/program/job6;

sleep 60;

# execute all the jobs related to vanilla_chocolate_swirl
execute_swirl.sh;

Thanks, Mike

possible duplicate of [How to wait in bash for several subprocesses to finish and return exit code !=0 when any subprocess ends with code !=0?](http://stackoverflow.com/questions/356100/how-to-wait-in-bash-for-several-subprocesses-to-finish-and-return-exit-code-0) — Wrikken, Aug 22 '14 at 20:32
I don't see what the problem is. You haven't mentioned anything about running any of these jobs in the background, so it should all be sequential. — glenn jackman, Aug 22 '14 at 20:35
Well if you are not running the script with "&" for background running then it should be straight forward. I don't understand the issue here. — dganesh2002, Aug 22 '14 at 20:36
The jobs are like this: the wrapper calls: execute_vanilla.sh; execute_chocolate.sh; execute_vanilla.sh contains: /usr/local/bin/job1 & /usr/local/bin/job2 & /usr/local/bin/job3; — Mike T, Aug 22 '14 at 20:44

dganesh2002 · Answer 1 · 2014-08-23T01:10:34.217

1

Use this logic ->

execute_vanilla.sh :

/usr/local/bin/program/job1 & 
pid_job1=$!
/usr/local/bin/program/job2 & 
pid_job2=$!
/usr/local/bin/program/job3 &
pid_job3=$!

wait $pid_job1 $pid_job2 $pid_job3

Or use below logic if you dont want to handle pids of each subtask. "jobs -p" gives PIDs of subprocesses that are in execution state.

/usr/local/bin/program/job1 & 
/usr/local/bin/program/job2 & 
/usr/local/bin/program/job3 &
.
.
/usr/local/bin/program/job40 &

for job in `jobs -p`
do
       wait $job
done

Similarly you can modify chocolate script. Then you can do following in your wrapper:.

./execute_vanilla.sh &
pid_vanilla=$!

./execute_chocolate.sh &
pid_chocolate=$!

wait $pid_vanilla $pid_chocolate

# execute all the jobs related to vanilla_chocolate_swirl
./execute_swirl.sh;

If you want chocolate script to wait for vanilla to finish then simply do it as follows. In that case you don't need to run any script in background inside a wrapper and flow will be sequential ->

./execute_vanilla.sh

./execute_chocolate.sh

# execute all the jobs related to vanilla_chocolate_swirl
./execute_swirl.sh;

edited Aug 23 '14 at 01:10

answered Aug 22 '14 at 22:03

dganesh2002

1,917
1
26
29

Thanks very much dganesh. By the way, I don't yet know how to make the lines in this comment box appear like code, line by line. I've a question, based on inferring (seeing) that 'wait' waits for the PID from each shell script before allowing progress in the wrapper. Since execute_chocolate.sh is dependent on execute_vanilla.sh completing, should the wrapper be like this: ./execute_vanilla.sh & pid_vanilla=$! wait $pid_vanilla ./execute_chocolate.sh & pid_chocolate=$! wait $pid_chocolate # execute all the jobs related to vanilla_chocolate_swirl ./execute_swirl.sh; – Mike T Aug 23 '14 at 00:38
dganesh, I just updated my original example. I added /usr/local/bin/**program**/job1. Does this change the way pid is used in your example? Thanks. – Mike T Aug 23 '14 at 00:48
@MikeT yes your logic should be fine as well but in that case no need to run anything in background inside a wrapper script. Check my update. I made the changes for the path updates as well. – dganesh2002 Aug 23 '14 at 00:54
Hi dganesh, an additional question: execute_vanilla.sh has over 40 jobs. Is there possibly another way to do this? Or need I do a pid_ for each job? Is the problem that I am putting each job in the background with '&'? I thought the '&' is to make multiple jobs run simultaneously. – Mike T Aug 23 '14 at 01:04
Well running in background is how the multiple jobs can be run simultaneously. I am editing it to make it easy for you. – dganesh2002 Aug 23 '14 at 01:07
Modifying your code to reference 'program' (as in reality the 'job' names are not tied to a root name but are mostly different names. I tried this but think it's not working: for program in `program -p` do wait $program done – Mike T Aug 23 '14 at 01:42
Thanks dganesh. hmmm....I set up 7 jobs followed by the 'for' loop you gave me. I sent an email to myself after the 'done' in the loop. What happened is that of jobs 1-7, jobs 1-6 finished. I got the email, and ps (and top) showed the 7th job still running. – Mike T Aug 23 '14 at 02:42
May be an OS is not starting the 7th process due to its own multithreading logic.. create a procedure for that for loop and call that procedure after every five jobs to be sure.. – dganesh2002 Aug 23 '14 at 02:56

David C. Rankin · Answer 2 · 2014-08-23T03:11:02.113

In addition to the original answer, the simple way to avoid asynchronous execution is don't background any of the jobs within execute_vanilla.sh or execute_chocolate.sh. For example, modify the scripts to contain:

execute_vanilla.sh contains:
    /usr/local/bin/program/job1
    /usr/local/bin/program/job2
    /usr/local/bin/program/job3

To make the successful completion of each job within execute_vanilla.sh a requirement before running the next command, you can modify execute_vanilla.sh as follows:

execute_vanilla.sh contains:
    /usr/local/bin/program/job1 && \
    /usr/local/bin/program/job2 && \
    /usr/local/bin/program/job3

The same would apply to execute_chocolate.sh. Either way, as long as you are not backgrounding any of the jobs or backgrounding vanilla or chocolate, then the execution of the scripts/jobs will be sequential. Making all jobs within vanilla or chocolate compound commands is also a further way of insuring sequential execution. To insure all commands are explicitly sequentially executed and also conditioned on the successful completion of prior commands you can call execute_vanilla.sh and execute_chocolate.sh as a compound command in your wrapper:

execute_vanilla.sh && execute_chocolate.sh

Absent making execute_vanilla.sh and execute_chocolate.sh a compound command, your wrapper script looks fine. Hope that helps.

dganesh and David, thanks very much! I am working on the implementation. Not sure yet if it's a successful one. Much appreciated. Mike — Mike T, Aug 23 '14 at 18:22

how to control the timing of call to scripts within a shell wrapper

2 Answers2