Parallel processing in bash with "&" and wait

Question

So I am trying to dump a lot of tables from 1 server and then restore them back in another server. I want to keep the dump going 1 by 1 and restore them as the dump finishes. I dont want to overwhelm the server by running more than 1 restore operation at the same time. So to achive that i am trying to have a third function which "wait" for a "restore function" to finish before calling another. But I am not able to use the "wait" correctly. The program is not waiting at all.

RESTORE(){

sleep 10 && echo "restore done" &

}


RESTORE_CALLER() {


echo "waiting for any restore with pid ${current_restore_pid}"
wait ${current_restore_pid}


echo "Calling restore"
RESTORE &
current_restore_pid=$!



}


DUMP(){

for ((i=0;i<5;i++));do
echo "dumping "

echo "restore caller"
RESTORE_CALLER &

done

} 

DUMP 

Answer 1

Just pipe it:

seq 1 5 |
while read l; do
    DUMP > "$l".dump
    echo "$l"
done |
while read l; do
    RESTORE < "$l".dump
    echo "$l"
done

But it's probably better to use another descriptor to transfer the data between the pipes, so that logs print nicely:

seq 1 5 |
while read l; do
   DUMP "$l"
   echo "$l" >&3
done 3> >(
    while read l; do
        RESTORE "$l"
    done
) |
cat

Example execution with two stubs:

DUMP() {
    sleep 0.$(($RANDOM % 10))
    echo "DUMPING $1"
}
RESTORE() {
    sleep 0.$(($RANDOM % 10))
    echo "RESTORING $1"
}

looks cool:

DUMPING 1
RESTORING 1
DUMPING 2
RESTORING 2
DUMPING 3
DUMPING 4
DUMPING 5
RESTORING 3
RESTORING 4
RESTORING 5

The | cat on the end is needed to synchronize the process substitution.

What is cool about it, you can use tools like xargs to easily parallelize the DUMP and RESTORE functions, ex run 3 DUMPs in parallel and 2 RESTORE in parallel:

DUMP() {
    echo "DUMPING $1"
    sleep 0.$(($RANDOM % 10))
    echo "DUMPED $1"
}
RESTORE() {
    echo "RESTORING $1"
    sleep 0.$(($RANDOM % 10))
    echo "RESTORED $1"
}

export -f DUMP RESTORE
seq 1 5 |
xargs -n1 -P3 bash -c 'DUMP "$1"; echo "$1" >&3' -- 3> >(
    xargs -n1 -P2 bash -c 'RESTORE "$1"' --
) | cat

And it looks even cooler:

DUMPING 1
DUMPING 2
DUMPING 3
DUMPED 3
RESTORING 3
DUMPING 4
DUMPED 4
RESTORED 3
RESTORING 4
DUMPING 5
DUMPED 1
RESTORING 1
RESTORED 4
DUMPED 2
RESTORING 2
DUMPED 5
RESTORED 2
RESTORING 5
RESTORED 1
RESTORED 5

Answer 2

I would guess that your script fails because every invocation of RESTORE_CALLER & creates a new process and a fresh current_restore_pid which is not shared.

If you split out the restore procedure into a separate script, you could use flock(1) to set an exclusive lock so that only one restore runs at a time (ordering is not guaranteed):

#!/bin/bash

lockfile=$(mktemp)
for ((i=0;i<5;i++)); do
    dump-function args
    flock -x "$lockfile" restore-program args &
done
rm "$lockfile"

Answer 3

Posting alternate answer, to allow for unlimited parallel dump, while allowing only one concurrent restore at any point.

Solution relatively straight forward: Dump jobs are forked, keeping track of each PID in array. Looping over restore job, and checking for each restore jobs that the corresponding dump jobs has completed, and that the previous restore jobs is also completed.

#! /bin/bash

RESTORE(){
        echo "Start Restore $1 - $$"
        sleep 10
        echo "End Restore $1 - $$"
}

DUMP() {
        echo "Start Dump $1 - $$"
        sleep 15
        echo "End Dump $1 - $$"
}


RUN_ALL(){

for ((i=0;i<5;i++));do
        DUMP $i &
        dump_pid[$i]=$!
done
restore_pid=
for ((i=0;i<5;i++));do
        wait ${dump_pid[i]}
        [ "$restore_pid" ] && wait $restore_pid
        RESTORE $i &
        restore_pid=$!
done

}

RUN_ALL

Answer 4

I believe the logic was correct, but the implementation has two places where tasks were sent to the background by mistake:

• In RESTORE, the '&' on 'sleep ...', make RESTORE return immediately.
• In DUMP, the '&' on 'RESTORE_CALLER ...', forks the restore into separate sub-shell, preventing wait from working (processes can only wait for direct children).

At the end, there is only one '&' - on the RESTORE operation, which is the only operation to go into the background.

RESTORE(){
# Remove backgroup
# sleep 10 && echo "restore done" &
sleep 10 && echo "restore done"
}


RESTORE_CALLER() {
echo "waiting for any restore with pid ${current_restore_pid}"
wait ${current_restore_pid}
echo "Calling restore"
RESTORE &
current_restore_pid=$!
}


DUMP(){
for ((i=0;i<5;i++));do
echo "dumping "

echo "restore caller"
# Remove Background
# RESTORE_CALLER &
RESTORE_CALLER
done
} 

DUMP