What is the correct sed in-place string replacement syntax?

Question

I'm trying to replace a line in a file and here is my current shell script:

sed -i "s%$line%$line_formatted%g" $file_source

Whenever I try to replace $line with $line_formatted I get this error:

sed: -e expression #1, char 81: unknown option to `s'

I was just wondering what the correct syntax would be?

Giving the below comment a try, it still doesn't replace the text. Here is the code I used:

echo "Here is line: "$line
echo "Here is line_formatted: "$line_formatted

# sed -i "s%$line%$line_formatted%g" $topicJRXML_file_source

awk -v old="$line" -v new="$line_formatted" '
s=index($0,old) { $0 = substr($0,1,s-1) new substr($0,s+length(old)) }
' $topicJRXML_file_source

cp "$topicJRXML_file_source" "$topicJRXML_file_destination"
echo "Here is line after awk: "$line

And this is my console output:

Here is line:  <property name="adhoc.display" value="Awk Test"/>
Here is line_formatted:  <property name="adhoc.display" value="$R{CUSTOM.Awk_Test.LABEL}"/>
Here is line after awk:  <property name="adhoc.display" value="Awk Test"/>

You are probably either running an old version of `sed` or one of `$line` or `$line_formatted` contains a `%` character. — Mort, Jan 05 '18 at 19:05
Do not use `sed` to modify xml. That's terribly fragile. Use an xml parser/writer instead. You can try `xmlstartlet` for example. — hek2mgl, Jan 06 '18 at 16:29
Using `echo` does not actually provide accurate, trustworthy information about your variables' values -- *especially* when you don't quote the expansions -- on that note, see [BashPitfalls #14](http://mywiki.wooledge.org/BashPitfalls#echo_.24foo). Use `bash -x yourscript`, or put `set -x` in the script, to enable much more reliable logging. — Charles Duffy, Jan 09 '18 at 21:52
`echo $var` is exactly the same between `var=$'hello\tworld'` and `var='hello world'`, for instance, despite them being two nonequal values. And if you have `var='*.txt'`, `echo $var` will list files in your current directory, instead of printing `*.txt` as a literal string. — Charles Duffy, Jan 09 '18 at 21:54
That said, if you were using the right tool for the job, you'd have something like `xmlstarlet ed -u '//property[@name="adhoc.display"]/@value' -v 'My Test' out.xml`. Might need to tweak it a bit if your document is namespaced -- there's not enough of it in the question to tell. — Charles Duffy, Jan 09 '18 at 21:55
BTW, `sed -i` has incompatible implementations between different major implementations (it's not in the POSIX standard for `sed` at all). What you gave here should work in simple cases for GNU `sed`, but certainly doesn't work with, for example, the BSD-derived `sed` distributed with MacOS. — Charles Duffy, Jan 09 '18 at 21:58
...to make `echo` *less* inaccurate (but not to make it sufficiently accurate to debug bizarre corner cases -- a task to which it's simply unsuited), one would want to put the variable expansions inside the quotes: `echo "Here is line: $line"`. Better for our case, though, is `printf '%q=%q\n' line "$line" line_formatted "$line_formatted"` — Charles Duffy, Jan 09 '18 at 22:00
...backing up my "don't ever use `echo`" ranting -- see [the POSIX specification for `echo`](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html), particularly the APPLICATION USAGE and RATIONALE sections (the former of which specifies outright: *New applications are encouraged to use printf instead of echo*). Note that behavior is utterly unspecified in a number of cases, including in any case where a literal backslash exists in a string, or when `-n` is given as the first argument. — Charles Duffy, Jan 09 '18 at 22:02

score 1 · Answer 1 · answered Jan 06 '18 at 14:18

1

See http://stackoverflow.com/q/29613304/1745001 for the horrendous task of doing what you want robustly with sed but instead of trying to force sed to pretend it's operating on literal strings when it doesn't support them, just use awk which does:

awk -v old="$line" -v new="$line_formatted" '
    s=index($0,old) { $0 = substr($0,1,s-1) new substr($0,s+length(old)) }
    { print }
' file

answered Jan 06 '18 at 14:18

Ed Morton

188,023
17
78
185

I tried using this awk replacement instead of sed but I am still unable to get the desired output. As you can see above the variable `$line` never actually got replaced by `$line_formatted`. Am I missing something? – Jason Smith Jan 09 '18 at 21:36
Yes, awk is a better choice than sed for manipulating literal strings -- but they're *both* the wrong tools for XML. – Charles Duffy Jan 09 '18 at 21:51
@JasonSmith Just like sed, awk prints it's output to stdout, not to the original file. Just like GNU sed has `-i` to overwrite the original fiule with it's output instead, GNU awk has `-i inplace` to so the same. With any tool you can instead do `tool 'script' file > tmp && mv tmp file` to update the original file with the tools output. In your question when you have `echo "Here is line after awk: "$line` you're just once again echoing the value of `line` that you set at the start of your script, it has nothing to do with the awk output. – Ed Morton Jan 09 '18 at 23:01
@CharlesDuffy the apparent XML that has now appeared in the question was added after I posted this answer. We still don't actually know if the OP is operating on XML or not but I agree his file names and text he's shared with us now do look suspicious! – Ed Morton Jan 09 '18 at 23:05
This is actually an XML document. @CharlesDuffy Since I'm running these scripts on a jenkins linux box I have to get approval from devops to install xmlstarlet. Right now I'm trying to achieve the most robust operation as possible for text replacement without xmlstarlet but if there is an awk/sed/perl replacement strategy that would be most preferred in my current scenario. – Jason Smith Jan 09 '18 at 23:36
Python has XML-aware tooling in the standard library -- would that do? (Ideally version 2.7 or something in the 3.x series so it has `xml.etree.ElementTree`; the older XML modules such as minidom are quite awful). – Charles Duffy Jan 10 '18 at 02:16
`xsltproc` is also available out-of-the-box in modern major distributions (it's fairly standard as build infrastructure, part of widely-used toolchains for building documentation). – Charles Duffy Jan 10 '18 at 02:35
Thanks for all the information! I will give all of this a shot, thanks @CharlesDuffy – Jason Smith Jan 10 '18 at 22:37

score 0 · Answer 2 · answered Jan 10 '18 at 02:34

Given the following XSLT template saved in the file update-adhoc-display.xslt:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:param name="newValue"/>

  <xsl:output omit-xml-declaration="yes" indent="yes"/>
  <xsl:strip-space elements="*"/>

  <xsl:template match="node()|@*">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*"/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="//property[@name='adhoc.display']/@value">
    <xsl:attribute name="value">
      <xsl:value-of select="$newValue"/>
    </xsl:attribute>
  </xsl:template>
</xsl:stylesheet>

...the following command:

xsltproc \
  --stringParam newValue "Updated Value" \
  update-adhoc-display.xslt in.xml >out.xml

...will transform an input document of:

<root>
  <property name="adhoc.display" value="Initial Value 1"/>
  <property name="other.content" value="Initial Value 2"/>
</root>

...to an output document of:

<root>
  <property name="adhoc.display" value="Updated Value"/>
  <property name="other.content" value="Initial Value 2"/>
</root>

_{Heavily inspired by https://stackoverflow.com/a/6873226/14122 by @Mithfindel, and -- as an answer to a known duplicate -- being flagged Community Wiki.}

What is the correct sed in-place string replacement syntax?

2 Answers2