Waiting for the last 10 parallel started jobs to finish in bash

Question

I have the following bash script, which starts a program several times in parallel and passes a control variable to each execution. The program is utilizes several resources, so after it was started 10 times in parallel I want to wait till the last 10 started, are finished.

I am currently doing this very roughly, by just waiting after 10 iterations for the longest time possible that 10 parallel started programs are finished.

Is there a straight forward way to implement this behavior?

steps=$((500/20))  
echo $steps
START=0

for((i=START;i < steps; i++))
do
        for((j=START;j < steps;j++))
        do
                for((k=START;k < steps;k++))
                do
                        n=$(($j*steps +$k))
                        idx=$(($i*$steps*$steps + $n))


                        if ! ((idx % 10)); then
                                echo "waiting for the last 10 programs"
                                sleep 10
                        else 
                                 ./someprogram $idx &
                        fi
                done
        done
done

If you want to track whether the programs succeeded or failed, you'll want to store their PIDs. An associative array mapping between PID and the `idx` is particularly useful, since that lets you know which `idx`s succeeded and failed. — Charles Duffy, Aug 11 '17 at 17:24
BTW, this approach as a whole is generally quite inefficient -- you usually have a bunch of wasted CPU time between when the first program in a batch finishes and when the last one does. Better off using something like `for ((k=start; k — Charles Duffy, Aug 11 '17 at 17:26
@CharlesDuffy Thank you, I saw that post and I am not quite sure if it is relatable for my case due to using an asynchronous device, which relies on the waiting time to finish up the jobs. However, does your answer already provide the wait for the program or is ist just for starting? — Kev1n91, Aug 11 '17 at 17:29
Which one? https://stackoverflow.com/questions/38160/parallelize-bash-script-with-maximum-number-of-processes calls for an answer that discusses `wait`, yes, and https://stackoverflow.com/questions/40377623/bash-wait-command-waiting-for-more-than-1-pid-to-finish-execution is *explicitly about* the `wait` command. As for "your answer", while I'm around in the bash tag a lot, I don't believe I have answers on either of those questions as of this writing. — Charles Duffy, Aug 11 '17 at 17:30

randomir · Accepted Answer · 2017-08-11T17:53:38.570

Well, since you already have a code in place to check the 10th iteration (idx % 10), the wait builtin seems perfect. From the docs:

wait: wait [-n] [id ...]

[...] Waits for each process identified by an ID, which may be a process ID or a job specification, and reports its termination status. If ID is not given, waits for all currently active child processes, and the return status is zero.

So, by waiting each time idx % 10 == 0, you are actually waiting for all previous child processes to finish. And if you are not spawning anything else than someprogram, then you'll be waiting for those (up to 10) last to finish.

Your script with wait:

#!/bin/bash

steps=$((500/20))  
START=0

for ((i=START; i<steps; i++)); do
    for ((j=START; j<steps; j++)); do
        for ((k=START; k<steps; k++)); do
            idx=$((i*steps*steps + j*steps + k))

            if ! ((idx % 10)); then
                wait
            else 
                ./someprogram $idx &
            fi
        done
    done
done

Also, notice you don't have to use $var (dollar prefix) inside arithmetic expansion $((var+1)).

I'm assuming above your actual script does some additional processing before calling someprogram, but if all you need is to call someprogram on consecutive indexes, 10 instances at a time, you might consider using xargs or GNU parallel.

For example, with xargs:

seq 0 1000 | xargs -n1 -P10 ./someprogram

or, with additional arguments to someprogram:

seq 0 1000 | xargs -n1 -P10 -I{} ./someprogram --option --index='{}' someparam

With GNU parallel:

seq 0 1000 | parallel -P10 ./someprogram '{}'

seq 0 1000 | parallel -P10 ./someprogram --option --index='{}' someparam

Waiting for the last 10 parallel started jobs to finish in bash

1 Answers1