Trouble replacing a single line in a file with multiple lines (BASH)

Question

Here's the setup, I have two variables:

$ echo "${ORIGINALSTRING}"
placeholdertext

$ echo "${REPLACEMENTSTRING}"
-----BEGIN CERTIFICATE-----
thisisarandombase64encodedstring
blahblahblahblahblahblahblahblah
blahblahblahblahblahblahblahblah
blahblahblahblahblahblahblahblah
blahblahblahblahblahblahblahblah
-----END CERTIFICATE-----

I need to replace the original string that exists inside of a text file with a string containing multiple lines. I ran into a dead end with sed:

$ sed -i "s/${ORIGINALSTRING}/${REPLACEMENTSTRING}/g" ./somedir/file.txt
sed: -e expression #1, char 48: unterminated `s' command

I tried maybe two dozen variations on that same command, but eventually I found out it was caused by one of the environment variables having multiple lines. So, sadly sed is out.

Does anyone have any other ideas on how to go about accomplishing this task? I heard that maybe awk can do what I want it to, but I haven't seen any good examples related to my specific use case.

Ed Morton · Answer 1 · 2021-06-26T12:57:01.977

3

ORIGINALSTRING="$ORIGINALSTRING" REPLACEMENTSTRING="$REPLACEMENTSTRING" awk '
    BEGIN {
        old = ENVIRON["ORIGINALSTRING"]
        new = ENVIRON["REPLACEMENTSTRING"]
    }
    s = index($0,old) {
        $0 = substr($0,1,s-1) new substr($0,s+length(old))
    }
    { print }
' file

The above will work no matter what characters are in either shell variable because it's using literal strings for the search and replace while a sed solution would fail if /, &, \1, ., *, ^, $, or various other characters were present in one or the other of them because sed doesn't have functionality to use literal strings, see Is it possible to escape regex metacharacters reliably with sed.

edited Jun 26 '21 at 12:57

answered Jun 26 '21 at 02:35

Ed Morton

188,023
17
78
185

@JohnKugelman using `-v` would break if either string contained escape sequences like `\t` which is why I'm using `ENVIRON[]` for literal strings instead. See [how-do-i-use-shell-variables-in-an-awk-script](https://stackoverflow.com/questions/19075671/how-do-i-use-shell-variables-in-an-awk-script). – Ed Morton Jun 26 '21 at 12:56
1

Oh yikes, I didn't know that. I've been using `-v` for years thinking it is safe. Thanks for the correction. [Further reading on Unix.SE](https://unix.stackexchange.com/a/113799/26300). – John Kugelman Jun 26 '21 at 13:02
You're welcome. Both approaches are "safe", but one is designed to interpret escape sequences while the other is designed to leave them literal so you just have to choose which to use depending on what your string can contain and what you want to happen to it. `-v` would probably be fine in this case but I don't know for sure what the OPs variables can contain so I'm erring on the side of caution and just going with literal strings. – Ed Morton Jun 26 '21 at 13:08

score 1 · Answer 2 · answered Jun 26 '21 at 02:38

I almost always recommend using ed to edit files, instead of trying to use the non-standard sed -i:

printf "%s\n" "g/$ORIGINALSTRING/c" "$REPLACEMENTSTRING" . w | ed -s input.txt

This changes every line that matches ORIGINALSTRING to the lines in REPLACEMENTSTRING (The . means end-of-input for what's known as insert mode), and writes the changed file back to disk.

The equivalent in sed would require adding a \\ to the end of every line to insert but the last:

sed -i "/$originalstring/c\\
${replacementstring//$'\n'/\\$'\n'}" input.txt

Also see Correct bash and shell script variable capitalization and think about if you should be using all upper case variable names or not.

score 0 · Answer 3 · answered Jun 26 '21 at 02:32

Replace string with multiline string with sed and bash and its Parameter Expansion:

ORIGINALSTRING='placeholdertext'

REPLACEMENTSTRING='-----BEGIN CERTIFICATE-----
thisisarandombase64encodedstring
blahblahblahblahblahblahblahblah
blahblahblahblahblahblahblahblah
blahblahblahblahblahblahblahblah
blahblahblahblahblahblahblahblah
-----END CERTIFICATE-----'

sed -i "s|${ORIGINALSTRING}|${REPLACEMENTSTRING//$'\n'/\\n}|" file

I switched from s/// to s|||.

Trouble replacing a single line in a file with multiple lines (BASH)

3 Answers3