Create new file from templates with bash script

Question

I have to create conf files and init.d which are very similar. These files permit to deploy new HTTP service on my servers. These files are the same and only some parameters change from one file to another (listen_port, domain, and path on server.)

As any error in these files leads to dysfunction of service I would like to create these files using a bash script.

For example:

generate_new_http_service.sh 8282 subdomain.domain.example /home/myapp/rootOfHTTPService

I am looking for a kind of templating module that I could use with bash. This templating module would use some generic conf and init.d scripts to create new ones.

Could I could use python templating engine?

Answer 1

You can do this using a heredoc. e.g.

generate.sh:

#!/bin/sh

#define parameters which are passed in.
PORT=$1
DOMAIN=$2

#define the template.
cat  << EOF
This is my template.
Port is $PORT
Domain is $DOMAIN
EOF

Output:

$ generate.sh 8080 domain.example

This is my template.
Port is 8080
Domain is domain.example

or save it to a file:

$ generate.sh 8080 domain.example > result

Answer 2

Template module for bash? Use sed, Luke! Here is an example of one of millions of possible ways of doing this:

$ cat template.txt 
#!/bin/sh

echo Hello, I am a server running from %DIR% and listening for connection at %HOST% on port %PORT% and my configuration file is %DIR%/server.conf

$ cat create.sh 
#!/bin/sh

sed -e "s;%PORT%;$1;g" -e "s;%HOST%;$2;g" -e "s;%DIR%;$3;g" template.txt > script.sh

$ bash ./create.sh 1986 example.com /tmp
$ bash ./script.sh 
Hello, I am a server running from /tmp and listening for connection at example.com on port 1986 and my configuration file is /tmp/server.conf
$ 

Answer 3

you can do this directly in bash, you do not even need sed. Write a script like that:

#!/bin/bash

cat <<END
this is a template
with $foo
and $bar
END

then call it like so:

foo=FOO bar=BAR ./template 

Answer 4

For simple file generation, basically doing

 . "${config_file}"
 template_str=$(cat "${template_file}")
 eval "echo \"${template_str}\""

would suffice.

Here ${config_file} contains the configuration variables in shell parseable format, and ${template_file} is the template file that looks like shell here document. The first line sources in the file ${config_file}, the second line puts the contents of the file ${template_file} into the shell variable template_str. Finally in the third line we build the shell command echo "${template_str}" (where the double quoted expression "${template_str}" is expanded) and evaluate it.

For an example of the contents of those two files, please refer to https://serverfault.com/a/699377/120756.

There are limitations what you can have in the template file or you need to perform shell escaping. Also if the template file is externally produced, then for security reasons you need to consider implementing a proper filtering prior to execution so that you will not for example lose your files when somebody injects the famous $(rm -rf /) in the template file.

Answer 5

Here's the approach that I ended up taking to solve this problem. I found it a little more flexible than some of the above approaches, and it avoids some of the issues with quotes.

fill.sh:

#!/usr/bin/env sh

config="$1"
template="$2"
destination="$3"

cp "$template" "$destination"

while read line; do
    setting="$( echo "$line" | cut -d '=' -f 1 )"
    value="$( echo "$line" | cut -d '=' -f 2- )"

    sed -i -e "s;%${setting}%;${value};g" "$destination"
done < "$config"

template:

Template full of important %THINGS%

"Note that quoted %FIELDS% are handled correctly"

If I need %NEWLINES% then I can add them as well.

config:

THINGS=stuff
FIELDS="values work too!"
NEWLINES="those\\nnifty\\nlinebreaks"

result: Template full of important stuff

"Note that quoted "values work too!" are handled correctly"

If I need those
nifty
linebreaks then I can add them as well.

Answer 6

[Edit] I changed my answer from the original one, that was years ago.

I like the answer from FooF above: https://stackoverflow.com/a/30872526/3538173

Yet, I prefer not to have an intermediary variable to store the whole content of the template file in memory.

. "${config_file}"
eval "echo \"$(cat "${template_file}")\""

Example

Create a template file. Let's call it example.tpl:

Hello, ${NAME}!
Today, the weather is ${WEATHER}. Enjoy!

Create a configuration file to store your variables. Let's call it good.conf:

NAME=John
WEATHER=good

Now, in the script where you want to render the template, you can write this:

#!/usr/bin/env bash

template_file=example.tpl
config_file=good.conf

. "${config_file}"
eval "echo \"$(cat "${template_file}")\""

# Or store the output in a file
eval "echo \"$(cat "${template_file}")\"" > out

You should see this wonderful output :)

Hello, John!
Today, the weather is good. Enjoy!

Caution with eval

When you use eval, if the template file contains some instructions, they will be executed, and it can be dangerous. For example, let's change the example.tpl above with this content:

Hello, ${NAME}!
Today, the weather is ${WEATHER}. Enjoy!

I'm a hacker, hu hu! Look, fool!
$(ls /)

Now, if you render your template file, you will see this:

Hello, John!
Today, the weather is good. Enjoy!

I'm a hacker, hu hu! Look, fool!
bin
boot
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

Now edit your file good.conf to have this content:

NAME=$(ls -l /var)
WEATHER=good

and render the template. You should see something like this:

Hello, total 8
drwxr-xr-x.  2 root root    6 Apr 11 04:59 adm
drwxr-xr-x.  5 root root   44 Sep 11 18:04 cache
drwxr-xr-x.  3 root root   34 Sep 11 18:04 db
drwxr-xr-x.  3 root root   18 Sep 11 18:04 empty
drwxr-xr-x.  2 root root    6 Apr 11 04:59 games
drwxr-xr-x.  2 root root    6 Apr 11 04:59 gopher
drwxr-xr-x.  3 root root   18 May  9 13:48 kerberos
drwxr-xr-x. 28 root root 4096 Oct  8 00:30 lib
drwxr-xr-x.  2 root root    6 Apr 11 04:59 local
lrwxrwxrwx.  1 root root   11 Sep 11 18:03 lock -> ../run/lock
drwxr-xr-x.  8 root root 4096 Oct  8 04:55 log
lrwxrwxrwx.  1 root root   10 Sep 11 18:03 mail -> spool/mail
drwxr-xr-x.  2 root root    6 Apr 11 04:59 nis
drwxr-xr-x.  2 root root    6 Apr 11 04:59 opt
drwxr-xr-x.  2 root root    6 Apr 11 04:59 preserve
lrwxrwxrwx.  1 root root    6 Sep 11 18:03 run -> ../run
drwxr-xr-x.  8 root root   87 Sep 11 18:04 spool
drwxrwxrwt.  4 root root  111 Oct  9 09:02 tmp
drwxr-xr-x.  2 root root    6 Apr 11 04:59 yp!
Today, the weather is good. Enjoy!

I'm a hacker, hu hu! Look, fool!
bin
boot
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
swapfile
sys
tmp
usr
var

As you can see, command injection in the configuration file and the template file is possible, and that's why you have to be extra careful:

• be sure of the content of the template file: check that there is NO command injection.
• be sure of the content of the configuration file: check that there is NO command injection as well. If the configuration file comes from someone else, you need to know and trust that person before rendering the template.

Imagine that you are a password-less sudoer, rendering the template file could result in ruining your system with a well-placed rm -rf.

As long as you control the content of these files, it is fine to use this eval templating.

If you have an external (untrusted) incoming configuration file, you should look for templating engine, that will isolate these kind of injection. For example, Jinja2 templating is quite famous in Python.

Answer 7

I ended up using envsubst which was available for me. It is widely available on most Linux systems in the gettext package. I used this because it will take care of selectively replacing variables and I don't have to escape them in the template as with e.g. eval. Here is an example.

$ cat dbtemplate.xml
<DB>
        <hostname>$DBHOST</hostname>
        <port>$DBPORT</port>
        <database>$DBNAME</database>
</DB>

$ export DBHOST=mydbhost1 DBPORT=1234 DBNAME=mydb; envsubst< dbtemplate.xml
<DB>
        <hostname>mydbhost1</hostname>
        <port>1234</port>
        <database>mydb</database>
</DB>

Answer 8

You can use python class string.Template

$ echo 'before $X after' > template.txt

$ python  -c 'import string; print(string.Template(open("template.txt").read()).substitute({"X":"A"}))'

before A after

or

$  python  -c 'import string, sys; print(string.Template(open("template.txt").read()).substitute({"X":sys.argv[1]}))' "A"

Here $X is a placeholder in the template and {"X":"A"} is a mapping of the placeholder to a value. In the python code we read the template text from the file, create a template from it, then substitute the placeholder with the command line argument.

Alternatively you can use Ruby's ERB, if Ruby is installed on your machine.

$ echo "before <%= ENV['X'] %> after" > template.txt

$ X=A erb template.txt

before A after

Here <%= ENV['X'] %> is a placeholder. ENV['X'] reads the value from the environment variable. X=A sets the environment variable to the desired value.

Answer 9

Elegant and short solution in one line with perl

I use perl to replace variables with their values:

export world=World beautiful=wonderful
echo 'I love you, $world! You are $beautiful.' >my_template.txt
perl -pe 's|\$([A-Za-z_]+)|$ENV{$1}|g' my_template.txt

The output: I love you, World! You are wonderful.

my_template.txt can contain variables prefixed with $.

Answer 10

use bash to generate bash script from template

this problem calls for a strict separation of variable data (script header) and constant code (script body)

#!/usr/bin/env bash

# expected result

# header
a=1
b=2

# body
echo "a=$a and b=$b"

in the template file sample.tpl.sh, i have only one template variable #%TEMPLATE_CONSTANTS

#!/usr/bin/env bash

# template file

# header
#%TEMPLATE_CONSTANTS

# body
echo "a=$a and b=$b"
echo "here in the script body,
#%TEMPLATE_CONSTANTS
is not replaced"

most environments lack a fixed-string-editor, so i made my own "fsed" with grep and dd, to replace only the first match of #%TEMPLATE_CONSTANTS

#!/usr/bin/env bash

# template processor

# https://stackoverflow.com/a/69479243/10440128
fixedReplaceFirst(){ # aka fsed (fixed string editor)
  tplFile="$1"
  pattern="$2"
  replace="$3"
  match="$(grep -b -m 1 -o -E "$pattern" "$tplFile")"
  offset1=$(echo "$match" | cut -d: -f1)
  match="$(echo "$match" | cut -d: -f2-)"
  matchLength=${#match}
  offset2=$(expr $offset1 + $matchLength)
  dd bs=1 if="$tplFile" count=$offset1 status=none
  echo -n "$replace"
  dd bs=1 if="$tplFile" skip=$offset2 status=none
}

read -d '' replace <<EOF
a=1
b=2
EOF

fixedReplaceFirst "sample.tpl.sh" "^#%TEMPLATE_CONSTANTS$" "$replace"

related

• Interpret as fixed string/literal and not regex using sed
• Substitution in text file **without** regular expressions
• fixed string editors
  • rpl.py
  • mysql's extra/replace
  • richardlloyd's replace

Answer 11

I suggest shtpl, a shell templating system: https://github.com/dontsueme/shtpl

It has a very simple syntax, is foolproof and does only need standard *nix tools.

Answer 12

Quick and elegant solution that safely supports full shell scripting.

tmpl.sh

#!/bin/sh

EOF=EOF
exec cat <<EOF | sh
cat <<EOF
$(cat $1 | \
    sed 's|\\|\\\\|g' | \
    sed 's|`|\\`|g' | \
    sed 's|\$|\\\$|g' | \
    sed "s|${OPEN:-<%}|\`eval echo |g" | \
    sed "s|${CLOSE:-%>}| 2>/dev/null \`|g")
$EOF
EOF

https://gitlab.com/risserlabs/community/cinch/-/blob/main/src/tmpl.sh

Simply run the following on any file you want to template.

sh tmpl.sh file.txt.tmpl

file.txt.tmpl

uname: <% $(uname) %>
pwd: <% `pwd` %>
ls: $(ls)
df: `df`
SHELL: <% $SHELL %>
HOME: <% ${HOME} %>
USER: $USER
PATH: ${PATH}

The output would be something like the following.

uname: Linux
pwd: /home/clayrisser
ls: $(ls)
df: `df`
SHELL: /usr/bin/zsh
HOME: /home/clayrisser
USER: $USER
PATH: ${PATH}

If you want to write the templated file, simply redirect the output to a new file.

sh tmpl.sh file.txt.tmpl > file.txt

You can change the open and close characters by setting OPEN and CLOSE.

file.txt.tmpl

uname: {{ $(uname) }}
pwd: {{ `pwd` }}
ls: $(ls)
df: `df`
SHELL: {{ $SHELL }}
HOME: {{ ${HOME} }}
USER: $USER
PATH: ${PATH}

OPEN="{{" CLOSE="}}" sh tmpl.sh file.txt.tmpl