How to expand shell variables in a text file?

Question

Consider a ASCII text file (lets say it contains code of a non-shell scripting language):

Text_File.msh:

spool on to '$LOG_FILE_PATH/logfile.log';
login 'username' 'password';
....

Now if this were a shell script I could run it as $ sh Text_File.msh and the shell would automatically expand the variables. What I want to do is have shell expand these variables and then create a new file as Text_File_expanded.msh as follows:

Text_File_expanded.msh:

spool on to '/expanded/path/of/the/log/file/../logfile.log';
login 'username' 'password';
....

---

Consider:

$ a=123
$ echo "$a"
123

So technically this should do the trick:

$ echo "`cat Text_File.msh`" > Text_File_expanded.msh

...but it doesn't work as expected and the output-file while is identical to the source.

So I am unsure how to achieve this.. My goal is make it easier to maintain the directory paths embedded within my non-shell scripts. These scripts cannot contain any UNIX code as it is not compiled by the UNIX shell.

Answer 1

This question has been asked in another thread, and this is the best answer IMO:

export LOG_FILE_PATH=/expanded/path/of/the/log/file/../logfile.log
cat Text_File.msh | envsubst > Text_File_expanded.msh

if on Mac, install gettext first: brew install gettext

see: Forcing bash to expand variables in a string loaded from a file

Answer 2

This solution is not elegant, but it works. Create a script call shell_expansion.sh:

echo 'cat <<END_OF_TEXT' >  temp.sh
cat "$1"                 >> temp.sh
echo 'END_OF_TEXT'       >> temp.sh
bash temp.sh >> "$2"
rm temp.sh

You can then invoke this script as followed:

bash shell_expansion.sh Text_File.msh Text_File_expanded.msh

Answer 3

If you want it in one line (I'm not a bash expert so there may be caveats to this but it works everywhere I've tried it):

when test.txt contains

${line1}
${line2}

then:

>line1=fark
>line2=fork
>value=$(eval "echo \"$(cat test.txt)\"")
>echo "$value"
line1 says fark
line2 says fork

Obviously if you just want to print it you can take out the extra value=$() and echo "$value".

Answer 4

If a Perl solution is ok for you:

Sample file:

$ cat file.sh
spool on to '$HOME/logfile.log';
login 'username' 'password';

Solution:

$ perl -pe 's/\$(\w+)/$ENV{$1}/g' file.sh
spool on to '/home/user/logfile.log';
login 'username' 'password';

Answer 5

One limitation of the above answers is that they both require the variables to be exported to the environment. Here's what i came up with that would allow the variables to be local to the current shell script:

#!/bin/sh
FOO=bar;
FILE=`mktemp`; # Let the shell create a temporary file
trap 'rm -f $FILE' 0 1 2 3 15;   # Clean up the temporary file 

(
  echo 'cat <<END_OF_TEXT'
  cat "$@"
  echo 'END_OF_TEXT'
) > $FILE
. $FILE

The above example allows the variable $FOO to be substituted in the files named on the command line. I'm sure it can be improved, but this works for me so far.

Thanks to both previous answers for their ideas!

Answer 6

If the variables you want to translate are known and limited in number, you can always do the translation yourself:

sed "s/\$LOG_FILE_PATH/$LOG_FILE_PATH/g" input > output

And also assuming the variable itself is already known

Answer 7

This solution allows you to keep the same formatting in the ouput file

Copy and paste the following lines in your script

cat $1 | while read line
do
  eval $line
  echo $line
  eval echo $line
done | uniq | grep -v '\$'

this will read the file passed as argument line by line, and then process to try and print each line twice: - once without substitution - once with substitution of the variables. then remove the duplicate lines then remove the lines containing visible variables ($)

Answer 8

Yes eval should be used carefully, but it provided me this simple oneliner for my problem. Below is an example using your filename:

eval "echo \"$(<Text_File.msh)\""

I use printf instead of echo for my own purposes, but that should do the trick. Thank you abyss.7 providing the link that solve my problem. Hope it helps.

Answer 9

Create an ascii file test.txt with the following content:

Try to replace this ${myTestVariable1} 
bla bla
....

Now create a file “sub.sed” containing variable names, eg

's,${myTestVariable1},'"${myTestVariable1}"',g;
s,${myTestVariable2},'"${myTestVariable2}"',g;
s,${myTestVariable3},'"${myTestVariable3}"',g;
s,${myTestVariable4},'"${myTestVariable4}"',g'

Open a terminal move to the folder containing test.txt and sub.sed.
Define the value of the varible to be replaced

myTestVariable1=SomeNewText

Now call sed to replace that variable

sed "$(eval echo $(cat sub.sed))" test.txt > test2.txt

The output will be

$cat test2.txt

Try to replace this SomeNewText 
bla bla
....

Answer 10

#logfiles.list:

$EAMSROOT/var/log/LinuxOSAgent.log
$EAMSROOT/var/log/PanacesServer.log
$EAMSROOT/var/log/PanacesStrutsGUI.log

#My Program:

cat logfiles.list | while read line
    do
        eval Eline=$line
        echo $Eline
    done