36

In Linux, say I have the following file (e.g. conf.properties):

HOST_URL=http://$HOSTNAME
STD_CONFIG=http://$HOSTNAME/config
USER_CONFIG=http://$HOSTNAME/config/$unconfigured

I want to create another file with all the environment variables replaced...e.g. say the environment variable $HOSTNAME is 'myhost' and $unconfigured is not set, a script should produce the following output:

HOST_URL=http://myhost
STD_CONFIG=http://myhost/config
USER_CONFIG=http://myhost/config/

I was thinking this could be done in a simple one-liner with some sort of sed/awk magic, but I'm no expert and my searches have been in vein, so appreciate any help.

Edit:

I should mention that the file can really be any format text file, for example xml. I just want to replace anything that looks like an env variable with whatever is currently set in the environment.

Andy Whitfield
  • 2,373
  • 2
  • 19
  • 22

9 Answers9

190

This is what envsubst is for.

echo 'Hello $USER'
Hello $USER
echo 'Hello $USER' | envsubst
Hello malvineous

You would probably use it more like this though:

envsubst < input.txt > output.txt

envsubst seems to be part of GNU gettext.

Malvineous
  • 25,144
  • 16
  • 116
  • 151
  • 9
    This seems to be the only really correct answer here. Should be marked up higher. – speedplane Feb 16 '17 at 06:06
  • 1
    I would just like to say thanks for this answer. My friend and I were struggling with sed, awk and other "tricks" and this answer saved us. Knowledge is power. Thank you! – William Martins Aug 24 '17 at 23:31
10
sed 's/$HOSTNAME/myhost/g;s/$unconfigured//g' yourfile.txt > another_file.txt

update:

Based on updates in your question, this won't be a good solution.

update2 :

This is based on an answer to a related question. I've hacked at it (I'm unfamiliar with perl) to remove undefined vars.

perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' yourfile.txt

Should work for any input text file, however you will need to define vars using the ${...} format which simplifies the string matching.

(rant regarding the evilness of eval moved to a separate post so as not to confuse readers)

Community
  • 1
  • 1
Shawn Chin
  • 84,080
  • 19
  • 162
  • 191
  • 1
    Or more readable (IMHO): `sed -e 's/$HOSTNAME/myhost/' -e 's/$unconfigured//` ... – DarkDust Mar 11 '11 at 15:00
  • Assuming I know the env variables used within the config files I could use this technique, but I really want something more generic. – Andy Whitfield Mar 11 '11 at 15:01
  • @Shawn-Chin Thanks! I'll probably use this as I like the `${}` style of using properties. – Andy Whitfield Mar 11 '11 at 15:23
  • 3
    I think the perl command can be simplified to `perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : ""/eg' yourfile.txt` based on comments on the related question, instead of having a 2nd regex to remove empty variables. You simply need to replace `$&` with `""` – Joel Pearson Feb 18 '13 at 06:38
  • @JoelPearson's regex has the advantage of not removing backslash – Kristo Aun Dec 12 '18 at 09:56
8

"eval is evil"

This is not an answer, but a warning in response to using eval for this task. You really really really don't want to do that.

Exhibit 1: a malicious template file:

HOST_URL=http://$HOSTNAME
STD_CONFIG=http://$HOSTNAME/config
USER_CONFIG=http://$HOSTNAME/config/$unconfigured
&& cat /etc/redhat-release

An unsuspecting user:

[lsc@aphek]$ cat somefile | while read line; do echo $(eval echo `echo $line`); done
HOST_URL=http://xyz
STD_CONFIG=http://xyz/config
USER_CONFIG=http://xyz/config/
Red Hat Enterprise Linux WS release 4 (Nahant Update 9)

Note the last line!

Now, imagine the possibilities....

Community
  • 1
  • 1
Shawn Chin
  • 84,080
  • 19
  • 162
  • 191
  • this came to mind too. i was surprised people considered that. there should be a substitution mechanism in bash, really! or else, we should write a standard tool for this. – Daren Thomas Oct 31 '12 at 08:25
6

I'd do it like this:

# Set the $HOSTNAME and other variables
# Now evaluate the properties file as a shell script.
. config.properties
# Write the values
cat >somefile <<EOF
HOST_URL=$HOST_URL
STD_CONFIG=$STD_CONFIG
USER_CONFIG=$USER_CONFIG
EOF

Edit: Or this very nasty thing (I'm sure there's a better way)

for name in HOST_URL STD_CONFIG USER_CONFIG
    echo "$name=$(eval echo `echo '$'$name`)" >>somefile
end
DarkDust
  • 90,870
  • 19
  • 190
  • 224
3

Thanks to @DarkDust I came up with this:

cat somefile | while read line; do echo $(eval echo `echo $line`); done > somefile.replaced
Andy Whitfield
  • 2,373
  • 2
  • 19
  • 22
  • And even a one-liner ;-) Now that looks like a solution, as long as you know that your input files will not have `$` except for variables. – DarkDust Mar 11 '11 at 15:25
  • 2
    PLEASE DON'T USE THAT. Pardon the raised voice, but this is one situation where screaming is warranted. See explanation in my answer. – Shawn Chin Mar 11 '11 at 15:27
  • @Shawn No probs - thanks for the warning. I'm going with the perl line - works like a charm. – Andy Whitfield Mar 11 '11 at 15:33
  • I was more worried about unsuspecting user that may come across it in the future. Posted warning as a separate item so as not to cause confusion. – Shawn Chin Mar 11 '11 at 15:36
  • This will omit spaces at the beginning of lines and also lines containing `#`. – Ahmad Ahmadi Jan 29 '20 at 10:46
2

I used this oneliner to replace ${VARIABLE} style variables in a file:

TARGET_FILE=/etc/apache2/apache2.conf; for VARNAME in $(grep -P -o -e '\$\{\S+\}' ${TARGET_FILE} | sed -e 's|^\${||g' -e 's|}$||g' | sort -u); do sed -i "s|\${$(echo $VARNAME)}|${!VARNAME}|g" ${TARGET_FILE}; done

I'm pretty sure someone can do this in 1/3rd of the length using awk… feel challenged! ;)

Elias Probst
  • 275
  • 1
  • 12
1

Here is a snippet of Javascript that I like to have around for solving this exact problem:

// A Javascript version of envsubst for our builds
// Purpose: replace all ocurrences of ${VAR} with the equivalent var from the environment from stdin
var readline = require('readline');

var rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
  terminal: false
});

const environment = process.env;

rl.on('line', function(line) {
  const newLine = line.replace(/\$\{([a-zA-Z0-9_]+)\}/g, function(_match, variable) {
    const envVar = environment[variable];
    return envVar ? envVar : '';
  });

  process.stdout.write(`${newLine}\n`);
});

Hopefully this helps somebody else.

Robert Massaioli
  • 13,379
  • 7
  • 57
  • 73
0

Here's a short one-liner that uses python's curly brace formatting to safely do the magic:

contents=\"\"\"`cat $file`\"\"\"; python -c "import os;print $contents.format(**os.environ)"
  • avoids evil eval
  • allows outputting curly braces: use {{ instead of {
  • no need to specify vars explicitly when calling the script

For example, given properties file settings.properties:

# my properties file
someVar = {MY_ENV_VAR}
curlyBraceVar = has {{curly braces}}

Then, do the substitution with:

$ export MY_ENV_VAR="hello"
$ file=settings.properties 
$ contents=\"\"\"`cat $file`\"\"\"; python  -c "import os;print $contents.format(**os.environ)"
# my properties file
someVar = hello
curlyBraceVar = has {curly braces}

A script is here: https://raw.githubusercontent.com/aneilbaboo/machome/master/bin/substenv

Aneil Mallavarapu
  • 3,485
  • 5
  • 37
  • 41
  • 1
    This does not avoid evil eval, it just obfuscates it. `echo '"""; print "arbitrary code execution"; """' > ~/myfile;` creates a file that causes some arbitrary code to be evaluated. – that other guy Apr 24 '15 at 19:51
0

if you have installed nodejs you can run

npx @utft/tt -e FOO=bar /path/to/input /path/to/output

or you can run it programmatically https://github.com/utftufutukgyftryidytftuv/tt

user12940451
  • 1