Using multiline variable in sed command

Question

I have a multiline variable that I captured from STDOUT.
I want to insert an echo command using this multiline variable to line 15 in another script (target).

#!/bin/bash
TEST=`cat foo`

echo "$TEST"

sed -i "15i echo \"$TEST\" > someotherfile" target  

Contents of foo :

apples
oranges
bananas
carrots

I thought the sed command read in line feeds, which I confirmed my foo has:

user@test$ cat foo | tr -cd '\n' | wc -c
4

When I run my test.sh script, I see what's in $TEST, but am getting an error for the sed command:

user@test$ ./test.sh
apples
oranges
bananas
carrots
sed: -e expression #1, char 18: unknown command: `o'

What am I doing wrong? Thanks in advance.

Answer 1

^{GNU sed is assumed, as implied by the syntax used in the question.}

#!/bin/bash

# Read contents of file 'foo' into shell variable $test.
test=$(<foo)

# \-escape the newlines in $test for use in Sed.
testEscapedForSed=${test//$'\n'/\\$'\n'}

sed -i "15i echo \"$testEscapedForSed\" > someotherfile" target

• Your problem was that passing multi-line strings to sed functions such as i (insert) requires the newlines embedded in those strings to be \-escaped, so that sed knows where the string ends and additional commands, if any, start.

  • A (nonstandard) parameter expansion is used to replace all newlines in $test with themselves prefixed by \, using ANSI C-quoted string $'\n' to generate actual newline chars.

• Also note:

  • I've renamed TEST to test, because all-uppercase shell-variable names should be avoided.

  • I've used modern command-substitution syntax $(..) in lieu of legacy syntax `...`.

  • $(<foo) is a slightly more efficient - although nonstandard - way of reading the content of a file at once.

Answer 2

Try:

Solution1:

awk 'NR==15{print;system("cat foo");next} 1' Input_file

No need to get the complete file into a variable, we could simply print it whichever line of Input_file you want to print it.

Solution2:

line=15; sed -e "${line}r foo" target

Or (in script mode)

cat script.ksh
line=15;
sed -e "${line}r foo" target

Where you could change the number of line where you want to insert the lines from another file.

Answer 3

The i command in sed inserts the lines of text that end with a newline, up until a line that doesn't end with a backslash. The a and c commands are similar. Classic sed doesn't like the first line to appear on the same line as the i command; GNU sed isn't as fussy.

If you were writing the command manually, you'd need to write:

15i\
echo "apples\
oranges\
bananas\
carrots" > someotherfile

At issue now is "how do you want to create this given the file foo contains the list of names?". Sometimes, using sed to generate the sed script is useful. However, it can also be intricate if you need to get backslashes at the ends of lines which are subject to an i (or a or c) command, and it is simpler to circumvent the problem.

{
    echo "15i\\"
    sed -e '1s/^/echo "/' -e 's/$/\\/' -e '$s/\\$/" > someotherfile/' foo
} | sed -f /dev/stdin target

GNU sed can read its script from standard input using -f -; BSD (macOS) sed doesn't like that, but you can use -f /dev/stdin instead (which also works with GNU sed), at least on systems where there is a /dev/stdin.

Answer 4

Interesting issue. As already mentioned the whole story for sed to be able to insert multiline text in another file is that this new multiline text must have actually literral \n for line breaks.

So we can use sed to convert real new line chars to literal \n:

$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
#Alternative: a=$(sed "s#[\]#\0n#g" <(sed 's#[\]$##' <(tr '\n' '\\' <file3)))
$ echo "$a"
apples\noranges\nbananas\ncarrots

How this translation works:
* First we replace all new lines with a single backslash using tr
* Then we remove the backslash from the end of the string
* Then we replace all other backslashes with backaslash and n char.

Since now variable $a contains literal \n between lines, sed will translate them back to actuall new lines:

$ cat file4
Line1
line2
line3
$ sed "2i $a" file4
Line1
apples
oranges
bananas
carrots
line2
line3

Result:
Mutliline replacement can be done with two commands:

$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
$ sed "2i $a" file4

sed 2i means insert a text before line2. 2a can be used in order to insert something after line2.

Remark:
According to this post which seems to be a duplicate, translation of new lines to literal \n seems that can be done with just :

a=$(echo ${a} | tr '\n' "\\n")

But this method never worked in my system.

Remark2:
The sed operation sed "2i $a" = insert variable $a before line 2 , can be also expressed as sed "1 s/.*/\0\n$a/" = replace all chars of first line with the same chars \0 plus a new line \n plus the contents of variable $a => insert $a after line1 = insert $a before line2.