sed substitute variable contains newline (preserve it)

Question

I have a multi-line string, downloaded from the Web:

toast the lemonade
blend with the lemonade
add one tablespoon of the lemonade
grill the spring onions
add the lemonade
add the raisins to the saucepan
rinse the horseradish sauce

I have assigned this to $INPUT, like this:

INPUT=$(lynx --dump 'http://example.net/recipes' \
     | python -m json.tool \
     | awk '/steps/,/]/' \
     | egrep -v "steps|]" \
     | sed 's/[",]\|^ *//g; $d')

At this point, $INPUT is ready for substitution into my target file as follows:

sed -i "0,/OLDINPUT/s//$INPUT/" /home/test_file

Of course, sed complains about an unterminated s command - herein lies the problem.

The current workaround I am using is to echo $INPUT prior to giving it to sed, but then the newlines are not preserved. echo strips newlines - which is the problem.

The correct output should maintain its newlines. How can sed be instructed to preserve the newlines?

Answer 1

The hacky direct answer is to replace all newlines with \n, which you can do by adding

| sed ':a $!{N; ba}; s/\n/\\n/g'

to the long command above. A better answer, because substituting shell variables into code is always a bad idea and with sed you wouldn't have a choice, is to use awk instead:

awk -i inplace -v input="$INPUT" 'NR == 1, /OLDINPUT/ { sub(/OLDINPUT/, input) } 1' /home/test_file

This requires GNU awk 4.1.0 or later for the -i inplace.

Answer 2

If you're using Bash, you can substitute \n for the newlines:

INPUT="${INPUT//
/\\n}"

If you don't like the literal linefeed in your parameter expansion, you might prefer

INPUT="${INPUT//$'\n'/\\n}"

Side note - you probably mean to change the matched lines to your input, not substitute each of them. In which case, you don't want to quote the newlines, after all...

Answer 3

To clean up your code some.

This:

lynx --dump 'http://somesite.net/recipes' | python -m json.tool | awk '/steps/,/]/' | egrep -v "steps|]" | sed 's/"//g' |sed 's/,//g' | sed 's/^ *//g' | sed '$d'

Can be replaced with this:

lynx --dump 'http://somesite.net/recipes' | python -m json.tool | awk '/]/ {f=0} f {if (c--) print line} /steps/{f=1} {gsub(/[",]|^ */,"");line=$0}'

It may be shorten more, but I do not now what this does: python -m json.tool

This:

awk '/]/ {f=0} f {if (c--) print line} /steps/{f=1} {gsub(/[",]|^ */,"");line=$0}'

Does:

1. Print line after pattern steps to line before ] - awk '/steps/,/]/' | egrep -v "steps|]"
2. Removes ", , and all space in front of all lines. - sed 's/"//g' |sed 's/,//g' | sed 's/^ *//g'
3. Then remove last line of this group. - sed '$d'

---

Example:

cat file
my data
steps data
 more
 do not delet this
hei "you" , more data
extra line
here is end ]
this is good

awk '/]/ {f=0} f {if (c--) print line} /steps/{f=1} {gsub(/[",]|^ */,"");line=$0}' file
more
do not delet this
hei you  more data

Answer 4

You'll want to use an editor instead of sed's substitution:

$ input="toast the lemonade
blend with the lemonade
add one tablespoon of the lemonade
grill the spring onions
add the lemonade
add the raisins to the saucepan
rinse the horseradish sauce"

$ seq 10 > file

$ ed file <<END
1,/5/d
1i
$input
.
w
q
END

$ cat file
toast the lemonade
blend with the lemonade
add one tablespoon of the lemonade
grill the spring onions
add the lemonade
add the raisins to the saucepan
rinse the horseradish sauce
6
7
8
9
10

Answer 5

Assuming your input JSON fragment looks something like this:

{ "other": "random stuff",
  "steps": [
    "toast the lemonade",
    "blend with the lemonade",
    "add one tablespoon of the lemonade",
    "grill the spring onions",
    "add the lemonade",
    "add the raisins to the saucepan",
    "rinse the horseradish sauce"
  ],
  "still": "yet more stuff" }

you can extract just the steps member with

jq -r .steps

To interpolate that into a sed statement, you'd need to escape any regex metacharacters in the result. A less intimidating and hopefully slightly less hacky solution would be to read static text from standard input:

lynx ... | jq ... |
sed -i -e '/OLDINPUT/{s///; r /dev/stdin' -e '}' /home/test_file

The struggle to educate practitioners to use structure-aware tools for structured data has reached epic heights and continues unabated. Before you decide to use the quick and dirty approach, at least make sure you understand the dangers (technical and mental).