I have been left to the painful task of making numerous "simple" shell scripts universal for some not so universal processes. I am being forced to use configuration files, or rather based on what (in some cases) is already implemented: including the config file in the script at run-time or using the source command.
I tried my best to stray away from that based on the numerous disagreements with it but I may have no choice. I need to be able to allow for things like wild cards, custom arrays and placing current variables in the next variable. My question is mostly which has the easiest answer.
1. Can I make my current function work to interpret the $vars from the config file?
2. Will it matter? Is is any more secure than just using source?
My config file looks like an INI file although I did not intend it that way because I do not care.
[production]
file_date="%Y%m%d" #WAAAA That's not how we format our dates :(
#file_date="gotcha rm testfile"
ssh_user=myuser
server=myserver.net
source_directory=/home/myuser/inb
destination_directory="ONE"
destination_directory="TWO"
destination_directory="THREE"
file_name="SOME_UNIQUE_CONSTANT_*_$file_date" #Or no date
zip_file="SLIGHTLY_DIFFERENT_$file_name.zip" #Or no zip
data_file="A_LITTLE_DIFFERENT_$file_name.dat" #Or no extension
#What to use (sftp|sftpg3|...)
#Everything in directory yes/no
#Everything named *_THIS_*
#Rename *_THIS_* to *_THAT_*
#Who the hell knows...
[testing]
file_date="%Y%m%d" #WAAAA That's not how we format our dates :(
ssh_user=jsharpe
server=myserver.net
file_name="TEST_FILE_$file_date"
zip_file="$file_name.zip"
data_file="$file_name.dat"
source_directory="/home/$ssh_user/$server/$file_name"
destination_directory="test1"
destination_directory="test2"
destination_directory="test3"
[log]
level=verbose
report_on_warning=false
report_on_error=true
report_email_list="myemail@myserver.net"
report_email_list="youremail@myserver.net"
This is my main function for reading in the variables it works well enough to prevent dumb input. I tried using declare first and then case with the same results. Everything is interpreted literally which is actually what I wanted in an attempt to protect my script from the config files it reads, however it is bittersweet. Also thanks to antonio for his post which put my in the right? direction.
Update 2: Modified case to use a function and declare instead of eval and searches the config line for any previous included variables. Made the search function recursive for n number of variables (shown in the source directory for testing). Declare has an option for arrays so long as my config just repeats the same variable. Now I just need to test wildcards from the config file. Please try this and or and let me know if you can think of any major issues or flaws with my methods. I do know most of the cases do not have the function call and there are some obvious spots I need to error out if there is a reference to a variable that has not been defined yet.
function var_in_var() {
var_in_var_match = "$1"
if [[ "$1" =~ "$" ]] ; then
for var in "${prev_case[@]}" ; do :
if [[ "$1" =~ "$var" ]] ; then
local var_val="${!var}"
var_in_var_match="${1//\$$var/$var_val}"
var_in_var "$var_in_var_match" || return
return 0
fi
done
fi
}
function read_config_file() {
#Problematic if user is not knowlegeable on how to correctly make / modify config files.
echo "Reading config file $config_file"
shopt -s extglob # Turn on extended globbing.
#temp_file=$"$file_basename_$RANDOM.tmp"
tr -d '\r' '\t' < "$config_file" > "$temp_file" #Remove DOS EOL characters and tabs
local line_num=1 #Counter in case we need to error out config line number
local section=""
while IFS='= ' read lhs rhs ; do :
if [[ "$lhs" =~ "["*"]" ]] ; then
lhs="${lhs%\]}" #Remove closing brackets
lhs="${lhs#\[}" #Remove opening brackets
section="$lhs"_
elif [[ ! $lhs =~ ^\ *# && -n $lhs ]] ; then
rhs="${rhs%%\#*}" # Del in line right comments (like this one)
rhs="${rhs%%*( )}" # Del trailing spaces
rhs="${rhs%\"*}" # Del closing string quotes
rhs="${rhs#*\"}" # Del opening string quotes
#declare -x "$section"_"$lhs"="$rhs" #Does not store $vars from config file
x="$section$lhs"
rhs="${rhs//$/\$$section}"
case $lhs in
"ssh_user" ) declare "$x=$rhs" ; prev_case+=("$x");;
"server" ) declare "$x=$rhs" ; prev_case+=("$x");;
"source_directory" ) var_in_var $rhs || return ; declare "$x"="${var_in_var_match:-$rhs}" ; prev_case+=("$x");;
"file_date" ) declare "$x=$(date +"$rhs")" ; prev_case+=("$x");;
"file_name" ) var_in_var $rhs || return ; declare "$x"="${var_in_var_match:-$rhs}" ; prev_case+=("$x");;
"zip_file" ) var_in_var $rhs || return ; declare "$x"="${var_in_var_match:-$rhs}" ; prev_case+=("$x");;
"data_file" ) var_in_var $rhs || return ; declare "$x"="${var_in_var_match:-$rhs}" ; prev_case+=("$x");;
"destination_directory" ) declare -a "$x+=($rhs)" ; prev_case+=("$x");;
"level" ) declare "$x"="$rhs" ; prev_case+=("$x");;
"report_on_warning" ) declare "$x"="$rhs" ; prev_case+=("$x");;
"report_on_error" ) declare "$x"="$rhs" ; prev_case+=("$x");;
"report_email_list" ) declare "$x"="$rhs" ; prev_case+=("$x");;
* ) echo "Unknown variable $lhs skipping..." ; exit 1 ;;
esac
elif [[ $lhs == "" ]] ; then
if ! [[ $rhs == "" ]] ; then
echo "Reading line $line_num from file failed. Variable name is null. Exiting read..." ; has_warnings=true ;
return 1
fi
elif [[ $rhs == "" ]] ; then
echo "Reading line $line_num from file failed. Variable value is null. Exiting read..." ; has_warnings=true ;
return 1
else
if [[ $lhs =~ ^\ *# ]] ; then
echo "Line $line_num is a comment, skipping..."
else
echo "Reading line $line_num from file failed. Exiting read..." ; has_warnings=true ;
return 1
fi
fi
line_num=$(($line_num + 1))
done < "$temp_file"
#source "somefile.config" <-- Bad idea
shopt -u extglob # Turn off extended globbing.
echo_out_production_variables
echo_out_testing_variables
return 0
}
function echo_out_testing_variables() {
echo "Testing SSH_User: $testing_ssh_user"
echo "Testing Server: $testing_server"
echo "Testing Source Directory: $testing_source_directory"
echo "Testing File Date: $testing_file_date"
echo "Testing File Name: $testing_file_name"
echo "Testing Zip File: $testing_zip_file"
echo "Testing Data File: $testing_data_file"
for destination in "${testing_destination_directory[@]}" ; do : ; echo "Testing Destination(s): $destination" ; done
}
function echo_out_production_variables() {
echo "Production SSH_User: $production_ssh_user"
echo "Production Server: $production_server"
echo "Production Source Directory: $production_source_directory"
echo "Production File Date: $production_file_date"
echo "Production File Name: $production_file_name"
echo "Production Zip File: $production_zip_file"
echo "Production Data File: $production_data_file"
for destination in "${production_destination_directory[@]}" ; do : ; echo "Production Destination(s): $destination" ; done
}
My script outputs.
-bash-3.2$ ./get_file.bash -f test.conf
Reading config file test.conf
Line 3 is a comment, skipping...
Line 13 is a comment, skipping...
Line 14 is a comment, skipping...
Line 15 is a comment, skipping...
Line 16 is a comment, skipping...
Line 17 is a comment, skipping...
Production SSH_User: myuser
Production Server: myserver.net
Production Source Directory: /home/myuser/inb
Production File Date: 20151211
Production File Name: SOME_UNIQUE_CONSTANT_*_20151211
Production Zip File: SLIGHTLY_DIFFERENT_SOME_UNIQUE_CONSTANT_*_20151211.zip
Production Data File: A_LITTLE_DIFFERENT_SOME_UNIQUE_CONSTANT_*_20151211.dat
Production Destination(s): ONE
Production Destination(s): TWO
Production Destination(s): THREE
Testing SSH_User: jsharpe
Testing Server: myserver.net
Testing Source Directory: /home/jsharpe/myserver.net/TEST_FILE_20151211
Testing File Date: 20151211
Testing File Name: TEST_FILE_20151211
Testing Zip File: TEST_FILE_20151211.zip
Testing Data File: TEST_FILE_20151211.dat
Testing Destination(s): test1
Testing Destination(s): test2
Testing Destination(s): test3
Update 1: I looked into eval which is not better than source however It lets me keep my current function set up. My case statements look like
"file_date" ) eval "$x"=$(date +"${rhs}") ;; "ssh_user" ) eval "$x"="${rhs}" ;; "data_file" ) eval "$x"="${rhs}" ;;However I broke it by adding this line to my config file:
data_file="gotcha rm testfile"I think this is closer to what I wanted as I have moderate control at this point over the string and could possible sanitize it before attempting to eval it. Just a possibility at this point.
[SPOILER]: What my simple 474 line script does...
(sftp|scp|connect direct|sftpg3|...) n number of files from one location to n locations. The fun starts when no one can agree on what their files are named, if it needs to be renamed before sending, if all files or just some should be sent, if it needs a date added, if it comes zipped and if it needs to be unzipped at the destination.
