sed with arbitrary variables that might contain slashes

Question

I'm trying to use sed in the following-ish way:

VAR=`echo $STRING | sed s/$TOKEN/$REPLACEMENT/`

Unfortunately, I've come upon a case where $REPLACEMENT might possibly contain slashes. This causes the bash to complain, as it (the shell) potentially expands it to something like this:

#given $VAR=I like bananas, $TOKEN=bananas, and $REPLACEMENT=apples/oranges
VAR=`echo I like bananas | sed s/bananas/apples/oranges/`

So now sed is given an invalid argument with too many /'s. Is there any good way to handle that?

You should replace old and obsoleted back-tics with parentheses `$(...)` example: `VAR=$(echo $STRING | sed s/$TOKEN/$REPLACEMENT/)` — Jotne, Feb 11 '14 at 05:53
possible duplicate of [sed command : How to use variable](http://stackoverflow.com/questions/19151954/sed-command-how-to-use-variable) — devnull, Feb 11 '14 at 06:47
It's worse than that. If $TOKEN contains RE metacharacters (`.`, `*`, etc.) then sed will evaluate those too. Is that what you want? — Ed Morton, Feb 11 '14 at 17:07

tripleee · Accepted Answer · 2014-02-11T05:15:44.407

3

You can use any separator you like. "s!$TOKEN!$REPLACEMENT!" and "s%$TOKEN%$REPLACEMENT%" are popular alternatives.

Of course, in the general case, if the input could contain any characters whatsoever, you're back to square one. You could switch to a language which doesn't mix code and data so frivolously... including, in fact, the shell itself;

echo "${VAR/$TOKEN/$REPLACEMENT}"

(This is a Bash extension, though. It is available in some other shells, but not in classic Bourne shell.)

edited Feb 11 '14 at 05:15

answered Feb 11 '14 at 05:09

tripleee

175,061
34
275
318

Another good option is non-printable ascii control characters that are especially unlikely to appear in strings: `printf -v foo 's\x01%s\x01%s\x01' "searchstring" "replacsestring"; echo "$string" | sed -e "$foo"` – that other guy Feb 11 '14 at 16:55
1

@thatotherguy intriguing idea, but I'm hesitant to recommend that, as it's bound to break on various `sed` implementations out there. – tripleee Feb 11 '14 at 17:08
I doubt it. This would both violate POSIX, and would be harder to implement in C than simply allowing it. – that other guy Feb 11 '14 at 19:01
At least on OSX I was unable to get it to work, although the stumbling point was that `printf` didn't seem to want to accept control characters in the form `\x01`. Using a literal control-A character allowed me to work around that. But anyway, it's going to be hard to come up with something completely portable (which would not be more convoluted than simply using e.g. Perl instead). – tripleee Feb 11 '14 at 19:23

score 1 · Answer 2 · answered Feb 11 '14 at 05:08

1

Here is the fix

VAR="I like bananas"
TOKEN="bananas"
REPLACEMENT="apples/oranges"
echo $VAR |sed "s@$TOKEN@$REPLACEMENT@"

I like apples/oranges

answered Feb 11 '14 at 05:08

BMW

42,880
12
99
116

Ed Morton · Answer 3 · 2014-02-11T17:29:51.307

You can't reliably use sed for this as:

you typically can't find a character that is guaranteed not to be in any of the $TOKEN or $REPLACEMENT strings, and
sed cannot search for a string - it ALWAYS searches for regular expressions and so any RE metacharacters in $TOKEN will be evaluated as such and you cannot reliably implement code to escape them (despite what many people have attempted).

So, just use awk:

VAR=$(echo "$STRING" | awk -v t="$TOKEN" -v r="$REPLACEMENT" 'idx=index($0,t) {$0 = substr($0,1,idx-1) r substr($0,idx+length(t))} 1')

That will work for absolutely any character in any of the 3 strings except a newline in $STRING.

Without the echo it will handle a newline in $STRING too:

VAR=$(awk -v s="$STRING" -v t="$TOKEN" -v r="$REPLACEMENT" '
BEGIN {
    if (idx = index(s,t))
        s = substr(s,1,idx-1) r substr(s,idx+length(t))
    print s
}')

sed with arbitrary variables that might contain slashes

3 Answers3

Linked