How to use sed to search and replace html code strings from awk input variables

Question

I have 6 text files with 10 columns and 19 rows. The first row of each text file contains an identical header (intentionally ignored by awk). I create the header as part of the table header for formatting purposes.

Example - foo1.txt (shortened to first 4 rows of made-up data):

H1 H2 H3 H4 H5 H6 H7 H8 H9 H10
1 2 3 4 5 6 7 8 9 10
2 3 4 5 6 7 8 9 10 11
3 4 5 6 7 8 9 10 11 12

A template html file holding some CSS, table positions, and table strings to be replaced. It is important that this template html file is not written over, as it must be used many times for other cases (other sets of 6 text files). The table strings (MYTABLE1, MYTABLE2, …) will be what needs to be replaced by the shell script.

Example - template.html:

    <!--some html and css code, followed by below code-->
    <div>
    <div class="wrap">
    <table>
    <caption>foo1</caption>
    <tbody>
    MYTABLE1
    </tbody>
    </table>
    </div>
    <div>
    <div class="wrap">
    <table>
    <caption>foo2</caption>
    <tbody>
    MYTABLE2
    </tbody>
    </table>
    </div>
    <div>
    <div class="wrap">
    <table>
    <caption>foo3</caption>
    <tbody>
    MYTABLE3
    </tbody>
    </table>
    </div>
    <!--then, continues through foo6 and MYTABLE6 and other html code-->

The bash script opens each text file, and using awk, creates the row and reads from the files to fill in each row below the header row. The table html is included between the values coming in from the text files. The output from awk is stored as a variable, which is then passed to sed to search the template.html file for the MYTABLE* strings, and replace them with the variables containing additional table code. Then, sed is to create a new html file so as to not write over the template.html file. The awk part of the script works as intended, however the sed part complains about the 's/ and fails. I presume this is because there is html code being passed? I have tried multiple ways to get sed to accept the string variable, with each attempt having the 's/ failure.

Example - make_table.sh (included only the first 3 table elements to create):

#!/bin/bash

STRING1=$(cat foo/foo1.txt | awk ' NR==1{next} BEGIN {
print "<tr><th class=\x22right\x22>H1</th>", "<th class=\x22right\x22>H2</th>", "<th>H3</th>", "<th>H4</th>", "<th>H5</th>", "<th>H6</th>", "<th>H7</th>", "<th>H8</th>",  "<th>H9</th>", "<th>H10</th></tr>" }
{ print "<tr><td class=\x22right\x22>" $1 "</td><td class=\x22right\x22>" $2 "</td><td>" $3 "</td><td>" $4 "</td><td>" $5 "</td><td>" $6 "</td><td>" $7 "</td><td>" $8 "</td><td>" $9 "</td><td>" $10 "</td></tr>" }')

STRING2=$(cat foo/foo2.txt | awk ' NR==1{next} BEGIN {
print "<tr><th class=\x22right\x22>H1</th>", "<th class=\x22right\x22>H2</th>", "<th>H3</th>", "<th>H4</th>", "<th>H5</th>", "<th>H6</th>", "<th>H7</th>", "<th>H8</th>",  "<th>H9</th>", "<th>H10</th></tr>" }
{ print "<tr><td class=\x22right\x22>" $1 "</td><td class=\x22right\x22>" $2 "</td><td>" $3 "</td><td>" $4 "</td><td>" $5 "</td><td>" $6 "</td><td>" $7 "</td><td>" $8 "</td><td>" $9 "</td><td>" $10 "</td></tr>" }')

STRING3=$(cat foo/foo3.txt | awk ' NR==1{next} BEGIN {
print "<tr><th class=\x22right\x22>H1</th>", "<th class=\x22right\x22>H2</th>", "<th>H3</th>", "<th>H4</th>", "<th>H5</th>", "<th>H6</th>", "<th>H7</th>", "<th>H8</th>",  "<th>H9</th>", "<th>H10</th></tr>" }
{ print "<tr><td class=\x22right\x22>" $1 "</td><td class=\x22right\x22>" $2 "</td><td>" $3 "</td><td>" $4 "</td><td>" $5 "</td><td>" $6 "</td><td>" $7 "</td><td>" $8 "</td><td>" $9 "</td><td>" $10 "</td></tr>" }')

echo $STRING1
#everything above works as intended

#I've tried (with no luck):
#sed -e 's/MYTABLE1/'${STRING1}'/' \
#sed -e 'c/MYTABLE1/'"$(echo ${STRING1})"'/' \

#below does not work
sed -e 's/MYTABLE1/'"$(echo ${STRING1})"'/' \
    -e 's/MYTABLE2/'"$(echo ${STRING2})"'/' \
    -e 's/MYTABLE3/'"$(echo ${STRING3})"'/' \
    < template.html > template_new.html

How can I get sed to accept those STRING* commands? Could this be done in pure awk (not sure if awk can read the template.html and write the output to template_new.html though). I'd really like to avoid a pure sed solution, as beyond simple string substitution, sed format makes no sense. Can I better optimize the awk code?

Do not use `cat` with `sed` or `awk`, they can read file directly: `STRING3=$( awk '{some awk code}' foo/foo3.txt)` — Jotne, Sep 19 '19 at 05:09

Graeme · Answer 1 · 2019-09-18T21:38:49.923

3

This is because you strings contain the / character which terminates the s command. However you don't have to use the / character to delimit the s command, sed will accept whatever follows the s. Try using a # instead:

sed -e "s#MYTABLE1#${STRING1}#"  \
    -e "s#MYTABLE2#${STRING2}#"  \
    -e "s#MYTABLE3#${STRING3}#"  \
    < template.html > template_new.html

Note I have also reduced the quoting and removed the echo commands which are not needed.

According to the POSIX specification, you can use any character as a delimiter for the s command other than a backslash or newline. Although GNU sed will even accept the backslash too. See: What delimiters can you use in sed?

edited Sep 18 '19 at 21:38

answered Sep 18 '19 at 21:32

Graeme

2,971
21
26

This is still not working with the pound or other delimiter characters I try. It has the same “unterminated s’ command” error. – user2030765 Sep 19 '19 at 16:58
@user2030765, how can you say that and post an answer below saying changing delimiters is the solution? Your answer has the closing parenthesis `)` missing from the command substitution, maybe that's why it didn't work...? – Graeme Sep 19 '19 at 22:19

score 0 · Answer 2 · answered Sep 19 '19 at 00:21

If you know the filenames of the data files at the time you create the template, which in your example it looks like you do, you can do this whole job with one awk process (and safely). Assume the template contains lines like MYTABLE foo1.txt that are to be substituted with everything else left unchanged -- and using some variables to make the code shorter and clearer:

awk <template >new '
BEGIN{ a="<tr>"; b="<th class=\x22right\x22>"; c="<th>"; d="</th>"; e="</tr>";
  h=a b "H1" d b "H2" d c "H3" d c "H4" d c "H5" d c "H6" d c "H7" d c "H8" d c "H9" d c "H10" d e;
  b="<td class=\x22right\x22>"; c="<td>"; d="</td>" }
$1=="MYTABLE" { f=$2; getline <f; print h; 
  while((getline <f)>0){ print a b $1 d b $2 d c $3 d c $4 d c $4 d c $4 d c $5 d c $6 d c $7 d c $8 d c $9 d c $10 d e };
  close(f); next }
{ print }'

# the close(f) is only required if you have too many files for awk to open concurrently,
# but good practice always

Also note it's possible to put awk's script in a file with -f instead of on the command line, but the functionality is the same either way.

user2030765 · Accepted Answer · 2019-09-20T16:46:49.907

0

The solution is to use what I had, and only change the delimiter to something other than a forward slash. The following works:

sed -e 's#MYTABLE1#'"$(echo ${STRING1}"'#'  \
    -e 's#MYTABLE2#'"$(echo ${STRING2}"'#'  \
    -e 's#MYTABLE3#'"$(echo ${STRING3}"'#'  \
    < template.html > template_new.html

edited Sep 20 '19 at 16:46

answered Sep 19 '19 at 17:30

user2030765

149
14

How to use sed to search and replace html code strings from awk input variables

3 Answers3