How to delay a command substitution in a shell script?

Question

I have a bash script that in a nutshell generates a file via other command (works fine) and then later in the script I do a wc command on the file generated. My problem is that i'm using command substitution on the wc command and when I execute the script it is executing this substitution immediately rather than waiting for the file to be generated earlier in the script. What are my options?

The script is a shell program to run Oracle SQLLoader to load data into an Oracle Table. The SQLLoader command generates a file with rejected records and I am trying to do a word count on it. This is my code:

#!/bin/sh

#   Set variables

program_name="My Program Name"
input_dir="/interface/inbound/hr_datafile"
log_dir="/interface/inbound/log"
bad_dir="/interface/inbound/hr_bad"
archive_dir="/interface/inbound/hr_archive"
input_base="Import_20171213"
input_ext="csv"

control_file="data_loader.ctl"

exit_status=0
d=`date "+%Y%m%d"`
t=`date "+%H%M"`


#   Check if data file exists, count records

if [ -f ${input_dir}/${input_base}*.${input_ext} ]; then
    data_file_name=`ls -1rt ${input_dir}/${input_base}*.${input_ext} | head -1 | cut -c 32-100`
    data_file_base=${data_file_name%.*}
    echo "Data file name: " ${data_file_name}
    echo "Data file base: " ${data_file_base}
    no_of_recs=`expr $(wc -l ${input_dir}/${data_file_name} | cut -c 1-8) - 1`
    echo "DEBUG no_of_recs: " ${no_of_recs}
    no_of_errs=0
else
    echo
    echo
    echo 
    echo "----------------------------- ${program_name} ------------------------------------------"
    echo "----------------------------------- Error report : ------------------------------------------------"
    echo 
    echo "Please place your Data files in the UNIX directory => "${input_dir}
    echo "${program_name} Process exiting ..."
    echo
    echo "---------------------------------------------------------------------------------------------------"
    echo
    echo
    echo
    echo
    echo 
    exit 1
fi


#   Run SQL*Loader

echo
echo "==> Loading Data...into MY_TABLE table"
echo

# Create a temporary control file to pass the data file name in
cat $XX_TOP/bin/${control_file} | sed "s/:FILENAME/${data_file_name}/g" > $XX_TOP/bin/${data_file_base}_temp.ctl

# NOTE:  The following sqlldr format is used to "hide" the oracle user name and password
sqlldr errors=100000 skip=1 control=$XX_TOP/bin/${data_file_base}_temp.ctl data=${input_dir}/${data_file_name} log=${log_dir}/${data_file_base}.log bad=${bad_dir}/${data_file_base}.bad <<-END_SQLLDR > /dev/null
apps/`cat ${DB_SCRIPTS}/.${ORACLE_SID}apps`
END_SQLLDR

exit_status=$?
echo "DEBUG exit_status " ${exit_status}    

# Remove temporary control file
rm -rf $XX_TOP/bin/${data_file_base}_temp.ctl


#   Check for Errors

if [ -f ${bad_dir}/${data_file_base}.bad ]; then
    echo
    echo "----------------------------- ${program_name} ------------------------------------------"
    echo "----------------------------------- Error report : ------------------------------------------------"
    echo
    grep 'Record' ${log_dir}/${data_file_base}.log > ${log_dir}/${data_file_base}.rec
    grep 'ORA' ${log_dir}/${data_file_base}.log > ${log_dir}/${data_file_base}.err
    paste ${log_dir}/${data_file_base}.rec ${log_dir}/${data_file_base}.err ${bad_dir}/${data_file_base}.bad
    echo
    echo "<---------------------------------End of Error Report---------------------------------------------->"
    echo

    # Count error records
    no_of_errs=$(wc -l ${bad_dir}/${data_file_base}.bad | cut -c 1-8)
    no_of_recs=$(expr ${no_of_recs} - ${no_of_errs})

    # Remove temp files
    rm ${log_dir}/${data_file_base}.rec
    rm ${log_dir}/${data_file_base}.err
    rm ${bad_dir}/${data_file_base}.bad
else
    echo "Bad File not found at ${bad_dir}/${data_file_base}.bad"
fi

if (( ${no_of_errs} > 0 )); then 
    echo "Error found in data file..."
    exit_status=1
else
    # Archive the data file if no errors
    mv ${input_dir}/${data_file_name} ${archive_dir}/${data_file_base}_$d"_"$t.${input_ext}
    echo "Data file archived to ${archive_dir}"
    exit_status=0
fi

echo
echo
echo 
echo "----------------------------- ${program_name} ------------------------------------------"
echo 
echo "Total records errored out         :" ${no_of_errs}
echo "Total records successfully read   :" ${no_of_recs}
echo "---------------------------------------------------------------------------------------------------"
echo




#   Final Exit Status



if [ ${exit_status} -eq 1 ]; then
    echo "==> Exiting process...Status : ${exit_status}"
    exit 1
fi

The file referenced in the if condition check is the file generated, so I'm checking if the file exists and then running the wc on it. I know that it's executing prematurely because this wc error appears in my script output before it should:

Data file name:  Import_20171213.csv
Data file base:  Import_20171213
DEBUG no_of_recs:  27

==> Loading Data...into MY_TABLE table

wc: 0653-755 Cannot open /interface/inbound/hr_bad/Import_20171213.bad.
Username:
SQL*Loader: Release 10.1.0.5.0 - Production on Thu Dec 21 12:42:39 2017

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Commit point reached - logical record count 27
Program exited with status 2

In the code, the wc command on the .bad file is performed after the sqlldr section, yet the log shows the error occurring before sqlldr is invoked.

Any ideas would be much appreciated! Thanks!

Answer 1

You aren't seeing all of the output you expect - anything after the call to SQL*Loader is missing - and the error from wc is coming in the wrong place, before instead of after the Username; prompt. But it shouldn't be erroring giving the construct you've used - if the file doesn't exist the if test should stop that line being reach.

The issue is that it isn't seeing the end of the heredoc. All of the commands beyond the start of the heredoc are being executed as part of the heredoc processing but aren't going anywhere or being displayed to the terminal (as expected), and something in the way that is all being evaluated is causing the wc to run unexpectedly, even though the file doens't exist. You're seeing the stderr output from that.

And that is all happening before SQL*Loader starts, so you see the Username: prompt from that afterwards.

From comments it seems the heredoc-ending END_SQLLDR is really indented in your actual code, which isn't reflected in the posted question. As you used the <<- heredoc form that implies it's indented with spaces rather than tabs. That is causing that to not be recognised as the end of the here doc.

From The Linux Documentation Project:

The closing limit string, on the final line of a here document, must start in the first character position. There can be no leading whitespace. Trailing whitespace after the limit string likewise causes unexpected behavior. The whitespace prevents the limit string from being recognized.

Removing the whitespace so that is the first thing on the line will fix it.