String expansion - escaped quoted variable to value

Question

To get started, here's the script I'm running to get the offending string:

# sed finds all sourced file paths from inputted file.
#
# while reads each match output from sed to $SOURCEFILE variable.
# Each should be a file path, or a variable that represents a file path.
# Any variables found should be expanded to the full path.
#
# echo and calls are used for demonstractive purposes only 
# I intend to do something else with the path once it's expanded.

PATH_SOME_SCRIPT="/path/to/bash/script"

while read -r SOURCEFILE; do
    echo "$SOURCEFILE"
    "$SOURCEFILE"
    $SOURCEFILE
done < <(cat $PATH_SOME_SCRIPT | sed -n -e "s/^\(source\|\.\|\$include\) //p")

You may also wish to use the following to test this out as mock data:

[ /path/to/bash/script ]

#!/bin/bash
source "$HOME/bash_file"
source "$GLOBAL_VAR_SCRIPT_PATH"

echo "No cow powers here"

For the tl;dr crew, basically the while loop spits out the following on the mock data:

"$HOME/bash_file"
bash: "$HOME/bash_file": no such file or directory
bash: "$HOME/bash_file": no such file or directory
"$GLOBAL_VAR_SCRIPT_PATH"
"$GLOBAL_VAR_SCRIPT_PATH": command not found
"$GLOBAL_VAR_SCRIPT_PATH": command not found

My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"? I should also state that although eval works I do not intend to use it because of its potential insecurities.

Protip that any variable value used in cat | sed would be available globally, including to the calling script, so it's not because the script cannot call the variable value.

FIRST SOLUTION ATTEMPT

Using anubhava's envsubst solution:

SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do 
    echo "$SOURCEFILE"
    envsubst <<< "$SOURCEFILE"; 
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")

This outputs the following:

"$SOMEVARIABLE"
""
"/home/nick/.another_file"
"/home/nick/.another_file"

Unfortunately, it does not expand the variable! Oh dear :(

SECOND SOLUTION ATTEMPT

Based upon the first attempt:

export SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do 
    echo "$SOURCEFILE"
    envsubst <<< "$SOURCEFILE"; 
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")
unset SOMEVARIABLE

which produces the results we wanted without eval and without messing with global variables (for too long anyway), hoorah!

Good runner-ups were further suggested using eval (although potentially unsafe) which can be found in this answer and here (link courtesy of anubhava's extended comments).

Please note if you read this question before this comment that I've changed two of the output errors in the third comment block as they were incorrect - apologies! — Nick Bull, Jun 30 '16 at 14:53
Thanks for updating the question with all the relevant details and suggested links +1 — anubhava, Jun 30 '16 at 16:01

score 1 · Accepted Answer · answered Jun 30 '16 at 14:57

1

My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"?

Yes you can use envsubst program, that substitutes the values of environment variables:

while read -r sourceFile; do
   envsubst <<< "$sourceFile"
done < <(sed -n "s/^\(source\|\.\|\$include\) //p" "$PATH_SOME_SCRIPT")

answered Jun 30 '16 at 14:57

anubhava

761,203
64
569
643

1

Works perfectly! Thank you for your help – Nick Bull Jun 30 '16 at 15:07
I'm afraid I have to take that back - this doesn't work 100%. Please check the updated question – Nick Bull Jun 30 '16 at 15:16
1

It will work, just add `export` before your variable declaration to make it available to environment. So make it `export SOMEVARIABLE="/home/nick/.some_path"` – anubhava Jun 30 '16 at 15:21
1

You are a god! Thank you, is there a way to do it without exporting them? Sometimes I just want the variables to be localised to a function. Apparently `export local variable=` is valid bash, but it is still available outside the scope of the function. – Nick Bull Jun 30 '16 at 15:22
In theory it might be possible but that solution will be without the simplicity of using `envsubst`. I will research if it is possible then I will update the answer. – anubhava Jun 30 '16 at 15:27
Unfortunately the library we are including this is is quite large and variable scope has been of issue before because of non-localised variables. Again, it's outside the scope (heh) of the question, but your help is greatly appreciated! – Nick Bull Jun 30 '16 at 15:28
1

[Take a look here](http://stackoverflow.com/a/20316582/548225) though I believe even that will be unsafe. – anubhava Jun 30 '16 at 15:41
1

Fantastic research - I've ran across similar solutions. I think for safety, `export VARIABLE` and then using `unset VARIABLE` once the operations are complete is the best method to meet all goals - the question will shortly be updated. Thank you! – Nick Bull Jun 30 '16 at 15:52

score 1 · Answer 2 · answered Jun 30 '16 at 14:57

1

I think you are asking how to recursively expand variables in bash. Try

expanded=$(eval echo $SOURCEFILE)

inside your loop. eval runs the expanded command you give it. Since $SOURCEFILE isn't in quotes, it will be expanded to, e.g., $HOME/whatever. Then the eval will expand the $HOME before passing it to echo. echo will print the result, and expanded=$(...) will put the printed result in $expanded.

answered Jun 30 '16 at 14:57

cxw

16,685
2
45
81

1

I'm only avoiding the downvote because I didn't specify no `eval`! I actually knew this method worked, but `eval` is extremely insecure and should be avoided at all costs as a solution in these kinds of scripts unless used very carefully - unfortunately it has not been used carefully enough in this answer to be considered safe IMO. Please read further here: http://stackoverflow.com/questions/17529220/why-should-eval-be-avoided-in-bash-and-what-should-i-use-instead – Nick Bull Jun 30 '16 at 15:06
1

Please note I've updated the answer to reflect the above comment - apologies for not doing that sooner. Upvoted as a correct-before-edit solution. – Nick Bull Jun 30 '16 at 15:09

String expansion - escaped quoted variable to value

2 Answers2