shell script to read values from one file and replace it in other file

Question

I would like to replace a set of strings into a new file. Source file file1.json values has to be replaced by values from file1.config into file file2.json. I have following script that is failing to do so.

file1.json

{
    "colorsArray":[{
            "red":"$ALERT_LEVEL",
            "green":"$NORMAL_LEVEL",
            "blue":"$RELAX_LEVEL"
        }
    ]
}

file1.config

[root@ip10]# cat file1.config
ALERT_LEVEL=DANGER
NORMAL_LEVEL=NORMAL
RELAX_LEVEL=RELAX

run.sh

    #!/bin/bash
set -x
if [ $# -ne 3 ]; then
  echo "USAGE:
        ./$0 file1.json file2.json file1.config"
  exit 1
fi
echo "#!/bin/bash
cat > $2 << EOF
`cat $1`
EOF" > $2;
chmod +x $2;
# load variables into env
. $3
# actually replace variables
./$2

Error:

[root@ip10]# ./run2.sh file1.json file2.json file1.config
./file2.json: line 11: warning: here-document at line 2 delimited by end-of-file (wanted `EOF')

file2.json appears, but it has no values replaced.

root@ip10]# cat file2.json 
    {
        "colorsArray":[{
                "red":"",
                "green":"",
                "blue":""
            }
        ]
    }

your here doc is damaged. check the manpage it needs to be sth like `cat << EOF > $2` if you want to redirect sth to a file. `cat >` will simply hang. and you need to escape the shebang's `!` probably. — Marc Bredt, Jan 10 '16 at 11:59
export vars, otherwise they aren't defined in a child process — user3159253, Jan 10 '16 at 12:16

Oleg Andriyanov · Accepted Answer · 2016-01-10T12:29:55.520

2

Just explaining the comment of @user3159253. Your file1.config should look like this for your code to work properly:

export ALERT_LEVEL=DANGER
export NORMAL_LEVEL=NORMAL
export RELAX_LEVEL=RELAX

P.S. IMHO the way you do the thing is a bit overelaborate. I'd prefer using sed to complete this task -- takes only one string:

sed file1.json -e 's/$ALERT_LEVEL/DANGER/g' -e's/$NORMAL_LEVEL/NORMAL/g' -e's/$RELAX_LEVEL/RELAX/g' >file2.json

edited Jan 10 '16 at 12:29

answered Jan 10 '16 at 12:23

Oleg Andriyanov

5,069
1
22
36

Thanks - I sourced the file and assumed as sufficient. – Chucks Jan 10 '16 at 12:28
@Chucks Please consider using `sed` as I mentioned in update. – Oleg Andriyanov Jan 10 '16 at 12:30
Looks like I spent too much time formatting. See my answer for the command `envsubst` as an alternative to `sed`, and the work-around to get the variables to be exported, without unnecessarily modifying/cluttering the syntax of the original config file. – michael Jan 10 '16 at 12:46
Exporting and assigning are two different things! You could `export ALERT_LEVEL NORMAL_LEVEL RELAX_LEVEL; source config_file` without adding `export` directive into config file. – F. Hauri - Give Up GitHub Jan 10 '16 at 12:48
@F.Hauri Fair play supposes you don't know what to export if you don't know the config file contents. Moving contents of the config file to the main script breaks modularity and makes config file useless. – Oleg Andriyanov Jan 10 '16 at 12:56
Also it should be noted that the syntax `export VAR=VALUE` is a bash-ism, widely supported but nevertheless not standard. E.g. `sash` in Debian and derivatives doesn't support it, demanding a separated form: `VAR=VAL; export VAR` – user3159253 Jan 10 '16 at 14:27

score 1 · Answer 2 · edited May 23 '17 at 12:15

This is the classic problem of reading env vars from a file, and substituting those values in another file. There is an inherent danger is using eval with your input file, so that should be avoided.

The simplest way is if your system already has the command envsubst, which does this for you. Usage:

$ var1=value1 var2=value2 envsubst < input.txt > output.txt

The trick here, though, is that your env vars are in another text file. So they aren't specified on the command line itself; they'll need to be exported to be seen by envsubst. E.g,

#!/bin/bash
export var1=value1
export var2=value2
envsubst < input.txt > output.txt

Here's one possible way to do this with an input file of env vars:

#!/bin/bash
eval $(sed 's/^.*=/export &/' file1.config)
envsubst < file1.json > file2.json

If you don't have envsubst available, then see this answer for other options (e.g., eval, sed, m4, ...): How to replace ${} placeholders in a text file?

shell script to read values from one file and replace it in other file

2 Answers2