Escape new lines from cat in sed expression

Question

This question from another user was wrongfully marked duplicate, so I'm posting it again, emphasising the problem.

I´m trying to set in one of my xml files a private key which is save in a an external repository for security reasons. This is my bash script

    sed -i -e "s/privateKey/`cat privatekey`/g" text.txt

Where privateKey is my key downloaded from a repository and text.txt contains

  <test>
     privateKey
  </test>

After run the script I was expecting this:

 <test>
      -----BEGIN RSA PRIVATE KEY-----
      privatekeyblablablablafoooo
      -----END-----------
 </test>

But because of the contains of the private key sed is getting me this error

    sed: 1: "s/privateKey/-----BEGIN ...": unescaped newline inside substitute pattern

Any idea how can I escape the new line characters, that are coming in from the cat command?

...btw, if you think a question should be reopened, you can just edit it -- that'll put it in the queue to be considered for reopening. That way we don't get a knowledge base cluttered up with different variants on the same things, the original person gets notice when there's an answer (and credit/repo for their question), &c. — Charles Duffy, Aug 01 '16 at 16:17

Lasse Meyer · Answer 1 · 2016-08-01T16:09:41.957

4

The solution is to simply replace every new line character with \\n beforehand:

cat privatekey | sed ':a;N;$!ba;s/\n/\\n/g'

Just put this command into your final sed command:

sed -i -e "s/privateKey/$(cat privatekey | sed ':a;N;$!ba;s/\n/\\n/g')/g" text.txt

The only thing left is to properly indent the file:

sed -i '3,4s/^/\t/' text.txt

Note: Apparently this isn't working on OSX, but it does work with Linux.

edited Aug 01 '16 at 16:09

answered Aug 01 '16 at 15:23

Lasse Meyer

1,429
18
38

it´s giving me exactly the same error :( I´ll check if I miss something else – paul Aug 01 '16 at 15:26
Are you using GNU sed? I just tried this again, it's working perfectly for me. – Lasse Meyer Aug 01 '16 at 15:49
I´m using osx but using -e should works, but nothing I still have the same error using sed -i -e "s/privateKey/$(cat privatekey | sed ':a;N;$!ba;s/\n/\\n/g')/g" text.txt finally I solved using sed -i -e '2r privatekey.txt' text.txt but I dont like at all have to provide in which line I wanted – paul Aug 01 '16 at 16:00
Since you're using Bash, you can use replacement in variable expansion to change newlines to escape sequences: `p=$( – Toby Speight Aug 02 '16 at 09:31

score 2 · Answer 2 · edited Apr 13 '17 at 12:36

2

A similar problem has been answered on unix.stackexchange - Substitute pattern within a file with the content of other file:

sed -i -e '/privateKey/ {' -e 'r privatekey' -e 'd' -e '}' text.txt

edited Apr 13 '17 at 12:36

Community

1
1

answered Aug 01 '16 at 15:43

Sundeep

23,246
2
28
103

it´s returning sed: 1: "/privateKey/ {": unexpected EOF (pending }'s) – paul Aug 01 '16 at 16:06
can you try the second option explained in that thread I've linked to? don't know the differences between GNU sed and the one in OSX – Sundeep Aug 01 '16 at 16:16

score 1 · Answer 3 · edited Apr 13 '17 at 12:36

1

replace-string-with-contents-of-a-file-using-sed

in OSX:

$ cat t.txt
<test>
privateKey
</test>

$ cat key.file 
-----BEGIN RSA PRIVATE KEY-----
privatekeyblablablablafoooo
-----END-----------

$ f1=$(<key.file)
$ f2=$(<t.txt)

$ echo "${f2//privateKey/$f1}"|tee t.txt
<test>
-----BEGIN RSA PRIVATE KEY-----
privatekeyblablablablafoooo
-----END-----------
</test>

in Linux:

$ sed -e '/privateKey/{r key.file' -e 'd}' t.txt
<test>
-----BEGIN RSA PRIVATE KEY-----
privatekeyblablablablafoooo
-----END-----------
</test>

$ sed -i -e '/privateKey/{r key.file' -e 'd}' t.txt
$ cat t.txt
<test>
-----BEGIN RSA PRIVATE KEY-----
privatekeyblablablablafoooo
-----END-----------
</test>

edited Apr 13 '17 at 12:36

Community

1
1

answered Aug 01 '16 at 15:43

Ronak Patel

3,819
1
16
29

it´s returning sed: 1: "/privateKey/{r privatek ...": unexpected EOF (pending }'s) I´m using OSX, are you using Linux? – paul Aug 01 '16 at 16:05
`sed -i -e` is a sure sign of an answer written for GNU sed (the BSD sed given in MacOS doesn't allow `-i` without an argument giving a suffix to be used for backup files). – Charles Duffy Aug 01 '16 at 16:23
yes `sed` is for Linux and `echo + variables` is for OSX - see updated answer – Ronak Patel Aug 01 '16 at 21:16

score 0 · Answer 4 · answered Aug 01 '16 at 16:20

The best answer is not to use sed for this at all. A gawk (GNU awk) function for the purpose is given in BashFAQ #21:

gsub_literal() {
  # STR cannot be empty
  [[ $1 ]] || return

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  gawk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  '
}

...which can be used as:

gsub_literal privateKey "$(<privatekey)"

Note the need for GNU awk -- the BSD awk shipped with MacOS will fail with awk: newline in string.

Escape new lines from cat in sed expression

4 Answers4