sed is replacing matched text with output of another command, but that command's output contains expansion characters

Question

I'm trying to replace text in a file with the output of another command. Unfortunately, the outputted text contains characters bash expands. For example, I'm running the following script to change the file (somestring references output that would break the sed command):

#!/bin/bash
somestring='$6$sPnfj/lnXwZVrec7$fCnL9uy1oWIMZduInKTHBAxhsQxGCsBpm2XfVFFqDPHKidrd93yfjbYvKgYexXHVcvkKdu9lbfy16Ek5GvKy/1'

sed '0,/^title/s/^title*/'"$somestring"'\n&/' $HOME/example.txt

sed fails with this error:

sed: -e expression #1, char 30: unknown option to `s'

I think bash is substuting the contents of $somestring when building the sed command, but is then trying to expand the resulting text. I can't put the entire sed script in single quotes, I need bash to expand it the first time, just not the second. Any suggestions? Thanks

Answer 1

here the forward slash / is the problem. If it's the only issue you can set sed to use a different delimiter.

for example

$ somestring="abc/def"; echo xxx | sed 's/xxx/'"$somestring"'/'
sed: -e expression #1, char 11: unknown option to `s'

$ somestring="abc/def"; echo xxx | sed 's_xxx_'"$somestring"'_'
abc/def

you also need to worry about & and \ chars and escape them if can appear in the replacement text.

If you can't control the the replacement string, either you have to sanitize with another sed script or, alternatively use r command to read it from a file. For example,

$ seq 5 | sed -e '/3/{r replace' -e 'd}'
1
2
3slashes///1ampersand&and2backslashes\\end
4
5

where

$ cat replace
3slashes///1ampersand&and2backslashes\\end

Answer 2

You have several errors here:

• the string somestring has characters that are significative for sed command (the most important being '/' that you are using as a delimiter) You can escape it, by substituting it with a previous

  somestring=$(echo "$somestring" | sed -e 's/\//\\\//g')

  that will convert your / chars to \/ sequences.

• you are using sed '0,/^title/s/^title*/'"$somestring"'\n&/' $HOME/example.txt which is looking to substitute the string titl followed by any number of e characters by that $somestring value, followed by a new line and the original one. Unfortunately, sed(1) doesn't allow you to use newline characters in the pattern substitution side of the s command, but you can afford the result by using the i command with a text consisting of you pattern (preceding any new line by a \ to interpret it as literal):

Finally the script leads to:

#!/bin/bash
somestring='$6$sPnfj/lnXwZVrec7$fCnL9uy1oWIMZduInKTHBAxhsQxGCsBpm2XfVFFqDPHKidrd93yfjbYvKgYexXHVcvkKdu9lbfy16Ek5GvKy/1'
somestring=$(echo "$somestring" | sed -e 's/\//\\\//g')
sed '/^title/i\
'"$somestring\\
" $HOME/example.txt

Answer 3

If your shell is Bash, you can use parameter substitution to replace the problematic /:

somestring="{somestring//\//\\/}"

That looks scary, but is easier to understand if you look at the version that replaces x with __:

somestring="${somestring//x/__}"

---

It might be easier to use (say) underscore as the delimiter for your sed s command, and then the substitution above would be

somestring="${somestring//_/\\_}"

---

If you already have backslashes, you'll need to first replace those:

somestring="${somestring//\\/\\\\}"
somestring="{somestring//\//\\/}"

If there were other characters that needed escaping (e.g. on the search side of s///), then you could extend the above appropriately.

Answer 4

This URL provides the cleanest answer:

Command to escape a string in bash

printf "%q" "$someVariable" 

will escape any characters you need escaped for you.