Wait for bash background jobs in script to be finished

Question

To maximize CPU usage (I run things on a Debian Lenny in EC2) I have a simple script to launch jobs in parallel:

#!/bin/bash

for i in apache-200901*.log; do echo "Processing $i ..."; do_something_important; done &
for i in apache-200902*.log; do echo "Processing $i ..."; do_something_important; done &
for i in apache-200903*.log; do echo "Processing $i ..."; do_something_important; done &
for i in apache-200904*.log; do echo "Processing $i ..."; do_something_important; done &
...

I'm quite satisfied with this working solution; however, I couldn't figure out how to write further code to be executed only once ALL of the loops have been completed.

Is there a way to do this?

Answer 1

There's a bash builtin command for that.

wait [n ...]
      Wait for each specified process and return its termination  sta‐
      tus.   Each  n  may be a process ID or a job specification; if a
      job spec is given, all processes  in  that  job’s  pipeline  are
      waited  for.  If n is not given, all currently active child pro‐
      cesses are waited for, and the return  status  is  zero.   If  n
      specifies  a  non-existent  process or job, the return status is
      127.  Otherwise, the return status is the  exit  status  of  the
      last process or job waited for.

Answer 2

Using GNU Parallel will make your script even shorter and possibly more efficient:

parallel 'echo "Processing "{}" ..."; do_something_important {}' ::: apache-*.log

This will run one job per CPU core and continue to do that until all files are processed.

Your solution will basically split the jobs into groups before running. Here 32 jobs in 4 groups:

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

To learn more:

• Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
• Walk through the tutorial (man parallel_tutorial). You command line will love you for it.

Answer 3

I had to do this recently and ended up with the following solution:

while true; do
  wait -n || {
    code="$?"
    ([[ $code = "127" ]] && exit 0 || exit "$code")
    break
  }
done;

Here's how it works:

wait -n exits as soon as one of the (potentially many) background jobs exits. It always evaluates to true and the loop goes on until:

1. Exit code 127: the last background job successfully exited. In that case, we ignore the exit code and exit the sub-shell with code 0.
2. Any of the background job failed. We just exit the sub-shell with that exit code.

With set -e, this will guarantee that the script will terminate early and pass through the exit code of any failed background job.

Answer 4

A minimal example with wait $(jobs -p):

  for i in {1..3}
  do
    (echo "process $i started" && sleep 5 && echo "process $i finished")&
  done  

  sleep 0.1 # For sequential output
  echo "Waiting for processes to finish" 
  wait $(jobs -p)
  echo "All processes finished"

Exemplary output:

process 1 started
process 2 started
process 3 started
Waiting for processes to finish
process 2 finished
process 1 finished
process 3 finished
All processes finished

Answer 5

This is my crude solution:

function run_task {
        cmd=$1
        output=$2
        concurency=$3
        if [ -f ${output}.done ]; then
                # experiment already run
                echo "Command already run: $cmd. Found output $output"
                return
        fi
        count=`jobs -p | wc -l`
        echo "New active task #$count:  $cmd > $output"
        $cmd > $output && touch $output.done &
        stop=$(($count >= $concurency))
        while [ $stop -eq 1 ]; do
                echo "Waiting for $count worker threads..."
                sleep 1
                count=`jobs -p | wc -l`
                stop=$(($count > $concurency))
        done
}

The idea is to use "jobs" to see how many children are active in the background and wait till this number drops (a child exits). Once a child exists, the next task can be started.

As you can see, there is also a bit of extra logic to avoid running the same experiments/commands multiple times. It does the job for me.. However, this logic could be either skipped or further improved (e.g., check for file creation timestamps, input parameters, etc.).