0

I am using the sed command on Ubuntu to replace content.

This initial command comes from here.

sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"

However, as you can see, I have a slash in the replacement. The slash causes the command to throw:

sed: -e expression #1, char 9: unknown option to `s'

Moreover, my replacement is stored in a variable.

So the following will not work because of the slash:

sed -i "$ s/$/ $1/" "$DIR./result/doc.md"

As stated here and in duplicate, I should use another delimiter. If I try with @:

   sed -i "$ s@$@ $1@" "$DIR./result/doc.md"

It gives the error:

sed: -e expression #1, char 42: unterminated `s' command

My question is:

How can I use a variable in this command as well as other delimiter than / ?

Itération 122442
  • 2,644
  • 2
  • 27
  • 73
  • `$@` has a meaning in bash. Use something else as the delimiter, or use e.g. Perl that comes with real variables instead of the shell's macro expansions. – choroba Mar 16 '20 at 14:58
  • 2
    You just picked a bad delimiter. `@` is a shell parameter, so `$@` inside double quotes is expanded. Build the `sed` script using a combination of single-quoted and double-quoted strings, or pick a different delimiter. Keep in mind, though, that without extra care, any value of `$1` could break the script. – chepner Mar 16 '20 at 14:58
  • Unrelated, consider using `ed` rather than `sed -i`. – chepner Mar 16 '20 at 14:59
  • @chepner using underscore (`sed -i "$ s_$_ $1_" "$DIR./result/doc.md"` gives the same error: sed: -e expression #1, char 38: unterminated `s' command – Itération 122442 Mar 16 '20 at 15:00
  • 1
    `_` is *also* a shell parameter :) Try `|`. – chepner Mar 16 '20 at 15:01
  • 3
    Does this answer your question? [Is it possible to escape regex metacharacters reliably with sed](https://stackoverflow.com/questions/29613304/is-it-possible-to-escape-regex-metacharacters-reliably-with-sed) – Wiktor Stribiżew Mar 16 '20 at 15:02
  • 1
    You can also escape the `$` in the script to avoid unintended parameter expansion regardless of your chosen delimiter: `"\$ s@\$@ $1@"`, for example. – chepner Mar 16 '20 at 15:03
  • @chepner Damn me ! Works like a charm now – Itération 122442 Mar 16 '20 at 15:29

3 Answers3

1

TL;DR:

Try:

sed -i '$ s@$@ '"$1"'@' "$DIR./result/doc.md"

Long version:

Let's start with your original code:

sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"

And let's compare it to the code you referenced:

sed -i '$ s/$/abc/' file.txt

We can see that they don't exactly match up. I see that you've correctly made this substitution:

file.txt --> "$DIR./result/doc.md"

That looks fine (although I do have my doubts about the . after $DIR ). However, the other substitution doesn't look great:

abc -->  /replacement

You actually introduced another delimeter /. However, if we replace the delimiters with '@' we get this:

sed -i '$ s@$@ /replacement@' "$DIR./result/doc.md"

I think that the above is perfectly valid in sed/bash. The $@ will not be replaced by the shell because it is single quoted. The $DIR variable will be interpolated by the shell because it is double quoted.

Looking at one of your attempts:

sed -i "$ s@$@ $1@" "$DIR./result/doc.md"

You will have problems due to the shell interpolation of $@ in the double quotes. Let's correct that by replacing with single quotes (but leaving $1 unquoted):

sed -i '$ s@$@ '"$1"'@' "$DIR./result/doc.md"

Notice the '"$1"'. I had to surround $1 with '' to basically unescape the surrounding single quotes. But then I surrounded the $1 with double quotes so we could protect the string from white spaces.

Mark
  • 4,249
  • 1
  • 18
  • 27
  • I tested it and it works. However I prefere the solution that has been provided to me in comments. This solution is just "replacing delimiters with pipes". `sed -i "$ s|$| $1|" "$DIR./result/doc.md"` – Itération 122442 Mar 16 '20 at 15:48
  • @FlorianCastelain, that only works until your input or output data has a pipe in it; it's not a complete and robust solution. Try where `$1` contains `'foo|bar|baz'`. – Charles Duffy Mar 16 '20 at 15:50
  • @FlorianCastelain: I do not like @ as a delimeter either. I typically use colon `:` when forward slash `/` becomes unwieldy. My point in answering is to illustrate how proper quoting can avoid the shell interpolation for $@. Meanwhile, quoting is flexible enough to indicate where you require interpolation. Nonetheless, I can't refute Charles' argument that a delimiter character in your `$1` variable will break your sed script. You can probably defend against it by searching $1 for characters that will break your script or going with approaches shared by Charles. – Mark Mar 16 '20 at 16:01
1

Don't use sed here; perl and awk allow more robust approaches.

sed doesn't allow variables to be passed out-of-band from code, so they always need to be escaped. Use a language without that limitation, and you have code that always works, no matter what characters your data contains.

The Short Answer: Using perl

The below is taken from BashFAQ #21:

inplace_replace() {
  local search=$1; shift; local replace=$1; shift
  in="$search" out="$replace" perl -pi -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' "$@"
}
inplace_replace '@' "replacement" "$DIR/result/doc.md"

The Longer Answer: Using awk

...or, using awk to do a streaming replacement, and a shell function to make that file replacement instead:

# usage as in: echo "in should instead be out" | gsub_literal "in" "out"
gsub_literal() {
  local search=$1 replace=$2
  awk -v s="${search//\\/\\\\}" -v r="${rep//\\/\\\\}" 'BEGIN {l=length(s)} {o="";while (i=index($0, s)) {o=o substr($0,1,i-1) r; $0=substr($0,i+l)} print o $0}'
}

# usage as in: inplace_replace "in" "out" /path/to/file1 /path/to/file2 ...
inplace_replace() {
  local search=$1 replace=$2 retval=0; shift; shift
  for file; do
    tempfile=$(mktemp "$file.XXXXXX") || { retval |= $?; continue; }
    if gsub_literal "$search" "$replace" <"$file" >"$tempfile"; then
      mv -- "$tempfile" "$file" || (( retval |= $? ))
    else
      rm -f -- "$tempfile" || (( retval |= $? ))
    fi
  done
}
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
0

Use shell parameter expansion to add escapes to the slashes in the variable:

$ cat file
foo
bar
baz

$ set -- ' /repl'

$ sed "s/$/$1/" file
sed: 1: "s/$/ /repl/": bad flag in substitute command: 'r'

$ sed "s/$/${1//\//\\\/}/" file
foo /repl
bar /repl
baz /repl

That is a monstrosity of leaning toothpicks, but it serves to transform this:

sed "s/$/ /repl/"

into

sed "s/$/ \/repl/"

The same technique can be used for whatever you choose as the sed s/// delimiter.

glenn jackman
  • 238,783
  • 38
  • 220
  • 352