Insert multiple lines and keep their indentation with sed

Question

I have some Tython functions that I want to insert in a file. Inserting multiple lines in itself works well using a variable and some \n, but the indentation isn't kept. Because it's Python code, that's a big issue, the code can't work as it is.

Here is what I tried:

cat sed-insertlines.sh

#!/bin/bash

read -r -d '' lines_to_insert << 'EOF'
def string_cleanup(x, notwanted):\n
    for item in notwanted:\n
        x = re.sub(item, '', x)\n
    return x\n
EOF

lines_to_insert=$(echo ${lines_to_insert} )

sed  -i "/import re  # Regular Expression library/a $lines_to_insert" sed-insertlines.txt

But here is what I get in the end when I cat sed-insertlines.txt:

#!/bin/python

import re  # Regular Expression library
def string_cleanup(x, notwanted):
 for item in notwanted:
 x = re.sub(item, '', x)
 return x


def string_replace(i_string, pattern, newpattern):
    string_corrected = re.sub(pattern, newpattern, i_string)
    return string_corrected

Lines are there but the indentation is gone.

Answer 1

First, let's get the data cleanly into a shell variable. Here's one way:

lines_to_insert=$(cat<<'EOF'
def string_cleanup(x, notwanted):
    for item in notwanted:
        x = re.sub(item, '', x)
    return x
EOF
)

Note that there are no \n added; you can just use the text you want to insert unmodified with the sole restriction that it can't contain a line consisting of exactly EOF (and if it does, you can change the here-doc delimiter.) Unfortunately, the later use of sed will modify the text by interpreting some backslash-sequences.

The correct syntax for the sed a command would be the following:

sed -i '/^import re/a \
def string_cleanup(x, notwanted):\
    for item in notwanted:\
        x = re.sub(item, '', x)\
    return x
'

(The commonly-seen sed 'a line to insert' is not Posix standard, and does not allow you to put leading spaces on the line. The correct syntax is as shown above; an a followed by whitespace, followed by a continuation marker and a newline.)

Note that every line except the last ends with a continuation marker (a trailing backslash). We could have put those in the text above, but that would defeat the goal of allowing you to use precisely the text you want inserted.

Instead, when we interpolate the shell variable into the sed command, we'll insert the backslashes using the global search-and-replace syntax:

# The following works with bash 4.3 and up
sed -i.bak "/^import re/a \
${lines_to_insert//$'\n'/$'\\\n'}
" sed-insertlines.txt

# Prior to v4.3, quoting worked differently in replacement
# patterns, and there was a bug with `$'...'` quoting. The
# following will work with all bashes I tested (starting with v3.2):
nl=$'\n' bsnl=$'\\\n'
sed -i.bak "/^import re/a \
${lines_to_insert//$nl/$bsnl}
" sed-insertlines.txt

---

Another solution is to use the mapfile command to read the lines into an array:

mapfile -t lines_to_insert <<'EOF'
def string_cleanup(x, notwanted):
    for item in notwanted:
        x = re.sub(item, '', x)
    return x
EOF

Now we can add the backslashes using printf:

sed -i.bak "/^import re/a \
$(printf '%s\\\n' "${lines_to_insert[@]}")
" sed-insertlines.txt

(The search-and-replace syntax would work on the array as well, but I think the printf command is more readable.)

Unfortunately, that adds an extra newline after the text because all of the lines in the original text were continued. If that's undesired, it could easily be removed in the second solution by inserting the backslash and newline at the beginning of the printf instead of the end, making a slightly less-readable command:

sed -i.bak "/^import re/a $(printf '\\\n%s' "${lines_to_insert[@]}")
" sed-insertlines.txt

---

Finally, based on a nice answer by Benjamin W, here's a version which uses the sed r command and process substitution (to avoid a temporary file):

sed '/^import re/r '<(cat<<'EOF'
def string_cleanup(x, notwanted):
    for item in notwanted:
        x = re.sub(item, '', x)
    return x
EOF
) sed-insertlines.txt

Answer 2

I would use the sed r command, which inserts the contents of a file after the current cycle:

#!/bin/bash

# Write code to be inserted into 'insertfile' with proper indentation
cat <<'EOF' > insertfile
def string_cleanup(x, notwanted):
    for item in notwanted:
        x = re.sub(item, '', x)
    return x
EOF

# Sed with r command
sed -i '/import re  # Regular Expression library/r insertfile' sed-insertlines.txt

# Remove temp file
rm -f insertfile

resulting in

import re  # Regular Expression library
def string_cleanup(x, notwanted):
    for item in notwanted:
        x = re.sub(item, '', x)
    return x


def string_replace(i_string, pattern, newpattern):
    string_corrected = re.sub(pattern, newpattern, i_string)
    return string_corrected

Answer 3

Awk solution for this in case you're interested :

python_file:

#!/bin/python

import re  # Regular Expression library

def string_replace(i_string, pattern, newpattern):
    string_corrected = re.sub(pattern, newpattern, i_string)
    return string_corrected

Our Script

#!/bin/bash
read  -rd '' lines_to_insert << 'EOF'
def string_cleanup(x, notwanted):
    for item in notwanted:
        x = re.sub(item, '', x)
    return x
EOF
awk -v from_shell="$lines_to_insert" '
{
if ($0 ~ /import re  # Regular Expression library/){
printf "%s\n%s\n",$0,from_shell
}
else{
print $0
}
}' python_file

Output:

#!/bin/python

import re  # Regular Expression library
def string_cleanup(x, notwanted):
    for item in notwanted:
        x = re.sub(item, '', x)
    return x

def string_replace(i_string, pattern, newpattern):
    string_corrected = re.sub(pattern, newpattern, i_string)
    return string_corrected

Note :

I have removed the \ns from the $lines_to_insert.