How to wait until all background jobs initiated in a while loop have all finished a certain steps?

Question

I'd like to start a sequence of background job in a while loop in a bash script:

END=10
i=0;
while [ $i -lt $END ]
do
(SOME COMMAND;\
sleep 3;
SOME COMMAND 1;\
SOME COMMAND 2;\
SOME COMMAND 3;) &
i=`expr $i + 1`
done

And outside this while loop I'd like to wait until all these 10 background job all finished SOME COMMAND 1 and then proceed, so the psudo code is something like:

until [the number of process that finished SOME COMMAND 1 is greater or equal to $END];
do
sleep 0.5
done

What I tried to do is to add another counter like this:

true_started_cnt=0
END=10
i=0;
while [ $i -lt $END ]
do
(SOME COMMAND;\
sleep 3;
SOME COMMAND 1;\
true_started_cnt=`expr $true_started_cnt + 1`;\
SOME COMMAND 2;\
SOME COMMAND 3;) &
i=`expr $i + 1`
done

until [ $true_started_cnt -ge $END ]
do
echo "Waiting for initial period for all streams to finish"
sleep 0.5
done

But this seems not working, possibly because of simultaneous write of the same global var by multiple bg jobs. I wonder how to achieve my intention in this piece of code.

Referring to this post, I'm also trying:

echo 1 >/dev/shm/foo

END=10
i=0;
while [ $i -lt $END ]
do
(SOME COMMAND;\
sleep 3;
SOME COMMAND 1;\
echo $(($(</dev/shm/foo)+1)) >/dev/shm/foo;\
SOME COMMAND 2;\
SOME COMMAND 3;) &
i=`expr $i + 1`
done

until [ $(echo $(</dev/shm/foo)) -ge $END ]
do
  echo "Waiting, true_started_cnt=$(echo $(</dev/shm/foo))"
  sleep 1
done

But still not working.

I guess you meant add `wait` outside the while loop? I think in that way I'm waiting for the whole bg job to finish, but my goal is just wait until they finish a sub-step, i.e. `SOME COMMAND 1` — IsaIkari, Jan 09 '22 at 12:57
Modifying variables in a subshell does not affect parent shell variables. — konsolebox, Jan 09 '22 at 14:16
You also need to lock /dev/shm. In my opinion all of these aren't needed. You just need to record child PIDs and check each if they're still active or not. — konsolebox, Jan 09 '22 at 14:21

score 1 · Accepted Answer · answered Jan 09 '22 at 14:02

There could be race condition in your last script as well, try this :

cp /dev/null /dev/shm/foo

END=10
for ((i=0; i<$END; i++)); do
    (
    echo SOME COMMAND;\
    sleep $((i+1));\
    echo $i SOME COMMAND 1;\
    echo $i >>/dev/shm/foo;\
    echo $i SOME COMMAND 2;\
    sleep $((i+1));\
    echo $i SOME COMMAND 3
    ) &
done

until [ $(wc -l /dev/shm/foo | awk '{print $1}') -ge $END ]
do
    echo "Waiting, /dev/shm/foo contains $(echo $(</dev/shm/foo))"
    sleep 1
done

echo "*********** All SOME COMMAND 1 finished"
wait
echo "Script finished"

score 0 · Answer 2 · answered Jan 09 '22 at 14:25

0

Recording progress isn't needed if all you want is to make sure all processes exit.

#!/bin/bash

END=10
pids=()

for (( i = 0; i < END; ++i )); do
    ( sleep "$(( RANDOM % 10 + 1 ))" ) &
    pids[$!]=$!
done


while [[ ${#pids[@]} -gt 0 ]]; do
    for pid in "${pids[@]}"; do
        echo "Waiting for $pid to exit."
        wait "$pid"
        kill -s 0 "$pid" >/dev/null || unset 'pids[pid]'
    done

    sleep 1
done

answered Jan 09 '22 at 14:25

konsolebox

72,135
12
99
105

I think in that way I'm waiting for the whole bg job to finish, but my goal is just wait until they finish a sub-step, i.e. SOME COMMAND 1 – IsaIkari Jan 09 '22 at 15:04

score 0 · Answer 3 · answered Jan 09 '22 at 14:35

You could use a FIFO queue to signal completion of a certain stage of jobs:

rendezvous.sh

#!/usr/bin/env bash

shopt -so errexit
shopt -so nounset

declare -ri job_count=5

# A FIFO queue where jobs will signal their completion.
mkfifo rendezvous

# Clean up FIFO queue on script exit.
trap EXIT EXIT; EXIT() {
  rm -f rendezvous
  trap  - EXIT      # Restore default interrupt handler.
  kill -s EXIT "$$" # Propagate interrupt.
}

# Wait until the known number of jobs have signaled their completion to the
# FIFO queue.
wait_rendezvous() {
  (
    local -i counter=0

    while read -r; do
      (( ++counter ))

      if (( counter == job_count )); then
        break
      fi
    done
  ) < rendezvous
}

# Signal completion to the FIFO queue.
show_at_rendezvous() {
  # For some reason, without the trailing ampersand some A jobs end up
  # dangling.
  echo "done" > rendezvous &
}

#
# Actual script logic starts below.
#
command_a() {
  local -i job_id=$1
  echo "Command A[$job_id]: start"
  sleep 5
  show_at_rendezvous
  echo "Command A[$job_id]: done"
}

command_b() {
  local -i job_id=$1
  echo "Command B[$job_id]: start"
  sleep 10
  echo "Command B[$job_id]: done"
}

# Launch jobs.
for i in $(seq 1 "$job_count"); do
  command_a "$i" &
  command_b "$i" &
done

# Wait for all A jobs to signal their completion.
wait_rendezvous
echo "All A commands done; now awaiting for B commands..."

# Wait for all remaining jobs to finish.
wait
echo "All B commands done."

Output

$ ./rendezvous.sh 
Command A[1]: start
Command B[1]: start
Command A[2]: start
Command B[2]: start
Command A[3]: start
Command B[3]: start
Command A[4]: start
Command B[4]: start
Command A[5]: start
Command B[5]: start
Command A[2]: done
Command A[1]: done
All A commands done; now awaiting for B commands...
Command A[5]: done
Command A[3]: done
Command A[4]: done
Command B[1]: done
Command B[2]: done
Command B[3]: done
Command B[4]: done
Command B[5]: done
All B commands done.

If we look at output, A3-5 ended after message "All A commands done". — Philippe, Jan 09 '22 at 15:45
@Philippe you're right, I should have added timestamps to the log. It's just a race condition in how the `echo` statements are executed, but the timeouts are respected. — Ionuț G. Stan, Jan 10 '22 at 12:20
I think it's because the `while` loop only runs once. Can you double check ? — Philippe, Jan 10 '22 at 12:22
@Philippe I've just checked by adding an echo statement of the counter value and it... varies :) I don't know much about the way Bash handles this kind of concurrency, but there are some race conditions in how output to stdout is processed. — Ionuț G. Stan, Jan 10 '22 at 12:27

How to wait until all background jobs initiated in a while loop have all finished a certain steps?

3 Answers3

rendezvous.sh

Output