How to safely handle "\n" with sed on automated scripts where "\n" can be in the input to these scripts?

Question

I have a shell script that is called via parameters (it's called by an external binary programm which I can not change), like this:

myscript.sh "param1" "param2"

Now, in this script there's a sed "s/param1/param2/"-like command involved and the param2 can contain literaly the newline escape sequence \n (like line1\nline2):

VAL=$(echo "$2" | sed -e 's/[\/&]/\\&/g')
sed -i "s/$1/$VAL/" /a/path/to/file

I already did this: Escape a string for a sed replace pattern to escape backslashes and ampersands that may occur, but this does not help handling the newline \n (sed ignores it).

I know how to do it manually in a script (by entering a real newline, pressing Return, in the shell script file at the according place or do some stuff like $(echo)), but I have no influence to the parameters that are passed.

How can I safely handle the newline sequence so that sed does its job and inserts a newline when \n occurs in the parameter?

How do you pass the parameter to sed? I did a quick test and for me no escaping is required. The script contains only on line `echo abcdefg | sed "s/$1/$2/"`. I called it like this `./script.sh "d" "\ntest\n"` which results in three lines as expected `abc`, `test`, and `efg`. — Rambo Ramon, Jun 29 '15 at 08:33
@RamboRamon I added the real part of the script to the question. — Foo Bar, Jun 29 '15 at 08:43
The `echo` seems to break it. If i do `echo "a\nb\nc"` I get three lines. Try invoking it with `-E` to suppress the interpretation. — Rambo Ramon, Jun 29 '15 at 08:49

score 0 · Answer 1 · answered Jul 02 '15 at 16:37

In this case, I would very strongly recommend replacing sed with perl. If you are able to do that, then your script becomes:

perl -pi -e 'BEGIN {$a=shift;$b=shift} s/$a/$b/' "$1" "$2" /a/path/to/file

You no longer need the VAL variable at all!

If for some bizarre reason you're absolutely restricted to sed, change the VAL= statement to:

VAL=$(echo "$2" | sed -ne '1h;2,$H;$x;$s/[\/&]/\\&/g;$s/\n/\\n/g;$p;')

But don't do that. Use the perl version instead.

score 0 · Answer 2 · answered Jul 02 '15 at 16:42

0

Replace \n with real newlines:

VAL=${VAL//\\n/$'\n'}

answered Jul 02 '15 at 16:42

chepner

497,756
71
530
681

score 0 · Answer 3 · answered Jul 02 '15 at 16:54

From BashFAQ #21, a generic string substitution tool that works with arbitrary literals (neither newlines nor regexp characters being special) using awk:

# usage: gsub_literal STR REP
# replaces all instances of STR with REP. reads from stdin and writes to stdout.
gsub_literal() {
  # STR cannot be empty
  [[ $1 ]] || return

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  '
}

Granted, that's a mouthful, but it's trivial to use:

gsub_literal "$1" "$val" <infile >outfile

score 0 · Answer 4 · answered Nov 18 '15 at 10:20

VAL=$(echo "$2" | sed -e 's/[\/&]/\\&/g')
How can I safely handle the newline sequence so that sed does its job and inserts a newline when \n occurs in the parameter?

You can just let sed undo the escaping of \n by adding s/\\n/\n/g, i. e.

 VAL=$(echo "$2" | sed -e 's/[\/&]/\\&/g;s/\\n/\n/g')

Test:

# set a 'line1\nline2'
# VAL=$(echo "$2" | sed -e 's/[\/&]/\\&/g;s/\\n/\n/g')
# sed "s/$1/$VAL/" <<<qay
qline1
line2y

How to safely handle "\n" with sed on automated scripts where "\n" can be in the input to these scripts?

4 Answers4

Linked