question on using bwait to wait for multiple bsub jobs to finish

Question

I am new to using LSF (been using PBS/Torque all along).

I need to write code/logic to make sure all bsub jobs finish before other commands/jobs can be fired.

Here is what I have done: I have a master shell script which calls multiple other shell scripts via bsub commands. I capture the job ids from bsub in a log file and I need to ensure that all jobs get finished before the downstream shell script should execute its other commands.

Master shell script

#!/bin/bash

...Code not shown for brevity..

"Command 1 invoked with multiple bsubs" > log_cmd_1.txt

Need Code logic to use bwait before downstream Commands can be used

"Command 2 will be invoked with multiple bsubs" > log_cmd_2.txt

and so on 

stdout captured from Command 1 within the Master Shell script is stored in log_cmd_1.txt which looks like this

Submitting Sample 101
Job <545> is submitted to .
Submitting Sample 102
Job <546> is submitted to .
Submitting Sample 103
Job <547> is submitted to .
Submitting Sample 104
Job <548> is submitted to .

I have used the codeblock shown below after Command 1 in the master shell script.

However, it does not seem to work for my situation. Looks like I would have gotten the whole thing wrong below.

while sleep 30m;
do
    #the below gets the JobId from the log_cmd_1.txt and tries bwait

    grep '^Job' <path_to>/log_cmd_1.txt | perl -pe 's!.*?<(\d+)>.*!$1!' | while read -r line; do res=$(bwait -w "done($line)");echo $res; done 1>
<path_to>/running.txt;
    # the below sed command deletes lines that start with Space
    sed '/^\s*$/d' running.txt > running2.txt;
    # -s file check operator means "file is not zero size"
    if [ -s $WORK_DIR/logs/running2.txt ]
        then
            echo "Jobs still running";
        else
            echo "Jobs complete";
            break;
    fi
done

The question: What's the correct way to do this using bwait within the master shell script.

Thanks in advance.

Answer 1

bwait will block until the condition is satisfied, so the loops are probably not neecessary. Note that since you're using done, if the job fails then bwait will exit and inform you that the condition can never be satisfied. Make sure to check that case.

What you have should work. At least the following test worked for me.

#!/bin/bash

# "Command 1 invoked with multiple bsubs" > log_cmd_1.txt
( bsub sleep 0; bsub sleep 0 ) > log_cmd_1.txt

# Need Code logic to use bwait before downstream Commands can be used
while sleep 1
do
    #the below gets the JobId from the log_cmd_1.txt and tries bwait

    grep '^Job' log_cmd_1.txt | perl -pe 's!.*?<(\d+)>.*!$1!' | while read -r line; do res=$(bwait -w "done($line)");echo "$res"; done 1> running.txt;
    # the below sed command deletes lines that start with Space
    sed '/^\s*$/d' running.txt > running2.txt;
    # -s file check operator means "file is not zero size"
    if [ -s running2.txt ]
        then
            echo "Jobs still running";
        else
            echo "Jobs complete";
            break;
    fi
done

Another way to do it. Which may is a little cleaner, is to use job arrays and job dependencies. Job arrays will combine several pieces of work that can be managed as a single job. So your

"Command 1 invoked with multiple bsubs" > log_cmd_1.txt

could be submitted as a single job array. You'll need a driver script that can launch the individual jobs. Here's an example driver script.

$ cat runbatch1.sh 
#!/bin/bash

# $LSB_JOBINDEX goes from 1 to 10
if [ "$LSB_JOBINDEX" -eq 1 ]; then

  # do the work for job batch 1, job 1

  ...

elif [ "$LSB_JOBINDEX" -eq 2 ]; then

  # etc
  ...

fi

Then you can submit the job array like this.

bsub -J 'batch1[1-10]' sh runbatch1.sh

This command will run 10 job array elements. The driver script's environment will use the variable LSB_JOB_INDEX to let you know which element the driver is running. Since the array has a name, batch, it's easier to manage. You can submit a second job array that won't start until all elements of the first have completed successfully. The second array is submitted with this command.

bsub -w 'done(batch1)' -J 'batch2[1-10]' sh runbatch2.sh

I hope that this helps.