Bash (parallel wait execution)

Question

I've got a master script that reads 5 lines from a file and executes 5 distinct shell scripts at a time and waits for 5 to finish before starting the next batch:

cat file |
while read line1 && read line2  && read line3 && read line4 && read line5 
sh script1.sh &
sh script2.sh &
sh script3.sh &
sh script4.sh &
sh script5.sh &
wait

Is there a way to ensure that there are always 5 scripts running at a time even if scripts {1,3} complete. ie if 1-5 are running concurrently and 1,2 finish. Start 6,7 immediately.

...or the GNU extension `xargs -P`, if you prefer C to Perl. — Charles Duffy, Nov 29 '17 at 01:56
line{1..5} are assigned as values for subsequent schema keys. ie schema1 = line1 and so on.... I then take these variables and build SQL output which i later execute for each distinct table. — manawa, Nov 29 '17 at 15:42

score 0 · Answer 1 · answered Nov 29 '17 at 01:50

0

Interesting question! The best way I can think of to do it is to keep an array of the processes' PIDs and spool up a new one if there are fewer than five in the array. Something like this:

while read line1 && read line2  && read line3 && read line4 && read line5 ; do
    sh script1.sh &
    PIDS+=($!)
    sh script2.sh &
    PIDS+=($!)
    sh script3.sh &
    PIDS+=($!)
    sh script4.sh &
    PIDS+=($!)
    sh script5.sh &
    PIDS+=($!)

    for PID in "${PIDS[@]}"; do
        if ! kill -0 "$PID"; then # If the PID is not alive
            if [ ${#PIDS[@]} -lt 5 ]; then
                echo "Process $PID dead, starting new one"
                sh scriptN.sh &
                PIDS+=($!)
            fi
        fi
    done
done

It still needs some work, but hopefully that gives you the idea. GNU Parrallel wouldn't be a bad call either if that's available.

answered Nov 29 '17 at 01:50

John Moon

924
6
10

boo, hiss re: all-caps variable names; see http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html, fourth paragraph, for relevant POSIX-specified conventions. (All-caps names are used for variables with meaning for the shell and OS; names with at least one lower-case character are reserved for application use and guaranteed not to conflict). – Charles Duffy Nov 29 '17 at 01:52
More on topic, consider `wait -n` (in bash 4.3 or later) to wait for exactly one process to exit without needing to iterate through the lot of 'em with `kill -0` to figure out exactly which one it is. – Charles Duffy Nov 29 '17 at 01:53
(Also, this question is very much a FAQ -- we get a new instance of it once a month at least). – Charles Duffy Nov 29 '17 at 01:55
...personally, I'm pretty fond of Chepner's answer at https://stackoverflow.com/a/38775799/14122, using `wait -n` with prompt-style expansion to retrieve the number of jobs in-line. – Charles Duffy Nov 29 '17 at 01:58
Thanks, Charles! I had thought all caps variables were conventional, but you've shown me the light. `wait -n` is also helpful here. Also, I didn't realize this was a FAQ - I've just recently started answering questions on here. Thanks for the feedback, cheers! – John Moon Nov 29 '17 at 01:59

Bash (parallel wait execution)

1 Answers1