script to log the completion of an LSF (bsub) job

Question

I have a script called by cron to run an LSF job.

I would like to be told when that job is submitted and when it completes. The-Powers-That-Be have decided to disable email notifications. So I am writing this script to output the relevant information to a logfile.

It almost works. Here is some code:

crontab:

00 12 * * * my_script.sh my_job.bsub

my_job.bsub:

#!/bin/bash
#BSUB -q my_queue
#BSUB -W 06:00

echo "I am a long process"
sleep 60

my_script.sh:

#!/bin/sh

BSUB_SCRIPT=$1

# run bsub_script (and get the LSF job id while you're at it)...
JOB_ID=`bsub < $BSUB_SCRIPT | awk -F[\<,\>] '{print $2}'`

# log that job was submitted...                
echo "`date +%Y-%m%d %T` submitted '$BSUB_SCRIPT' [$JOB_ID]" >> $HOME/my_logfile.txt

# and log when job completes...
bsub -w "ended($JOB_ID)" << EOF
#!/bin/bash
#BSUB -q my_queue
#BSUB -W 00:30

echo "`date +%Y-%m-%d %T` completed '$BSUB_SCRIPT' [$JOB_ID]" >> $HOME/my_logfile.txt

EOF

(I found this answer helpful in figuring out how to submit a job that waits until an earlier one has completed.)

The issue is that the 2nd call to date, in the heredoc, gets evaluated immediately, so I wind up with a logfile that looks like this:

my_logfile.txt:

2018-01-30 13:15:14 submitted 'my_job.bsub' [1234567]
2018-01-30 13:15:14 completed 'my_job.bsub' [1234567]

Notice how the times are exactly the same.

How can I ensure that evaluation of the content of the heredoc is deferred until the LSF job runs?

As is my custom, the answer comes to me once I ask my question: I can do it using 2 scripts. So instead of a heredoc, the final bsub line would be something like: `bsub -w "ended($JOB_ID)" my_other_script.sh "completed '$BSUB_SCRIPT' [$JOB_ID]"` And _that_ script would call date and output to the logfile. — trubliphone, Jan 30 '18 at 14:00

Squirrel · Accepted Answer · 2018-02-01T18:58:54.150

The date command in the heredoc is being expanded before being passed to bsub. You need to quote the EOF in your heredoc expression or escape the date command. See the answer to this question: How does "cat << EOF" work in bash?

In particular:

The format of here-documents is:
      <<[-]word
              here-document
      delimiter
...

If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion.

So, for example when I run

$ cat << EOF
> echo `date`
> EOF

The output is

echo Tue Jan 30 11:57:32 EST 2018

Note that the date command is expanded, which is what's happening in your script. However, if I quote the delimiter in the heredoc:

$ cat << "EOF"
> echo `date`
> EOF

You get the unexpanded output you want:

echo `date`

Similarly, escaping date would preserve the other variables you want to expand:

$ cat << EOF
> echo \$(date)
> EOF

Output:

echo $(date)

Thanks. That pointed me in the right direction. Because I use other script variables in the heredoc, I wound up _not_ quoting ''EOF'' and escaping the "date" function as follows: ```echo "\$(date +%Y-%m-%d %T) completed '$BSUB_SCRIPT' [$JOB_ID]" >> $HOME/my_logfile.txt``` Yours is still a correct answer. — trubliphone, Jan 31 '18 at 10:10
Sure, I updated the answer to include that solution as well. — Squirrel, Feb 01 '18 at 18:59

script to log the completion of an LSF (bsub) job

1 Answers1