linux shell: sed regex working with special char

Question

In my directory, there are some html files contains such a string:

sceneFilePath: "./video/video/960.mp4",

What I need to do is to replace the path above with the right path. So I write a script to do that:

find ./video -type f -name "*.mp4" -print0 | while IFS= read -r -d '' myfile; do
    tmp=$(basename "$myfile")   #example.mp4
    tmp="${tmp/.mp4/.html}"
    # Here I create a file named $tmp according to a template with the command `cp`
    cp -rf index.html "$tmp"
    sed -i '' "s#sceneFilePath:.*,#sceneFilePath: \"${myfile}\",#g" "$tmp"
done

But it doesn't work.

Here is an example of $myfile:
./video/Bentota & Hikkaduwa/Hotels/River House/River House - Balapitiya.mp4

It seems that it is because of the $myfile, which contains some special characters, such as , &, -, or that it is because the .* can't match ./video/video/960.mp4.

@anubhava I don't understand your question. `sed` is working on the copied html file. — Yves, May 09 '17 at 16:41
Concerning `.*`: I tried this: `echo "Hello, hello, hello, hello" | sed "s#ello.*,#ola,#g"`. The result is `Hola, hello` (because the `.*` is greedy). — Scheff's Cat, May 09 '17 at 16:53
You're going to continuously run into problems using sed and shell variables. Just use awk or perl and pass the variable to it. — 123, May 09 '17 at 17:23
If you're going to use sed, then do what it says in the answers to http://stackoverflow.com/q/29613304/1745001. Note that contrary to the answers you've got do far, `&` is not your only problem character. Alternatively just use awk since awk can operate on literal strings and then you don't have to worry about any of it. — Ed Morton, May 09 '17 at 21:15

score 1 · Answer 1 · answered May 09 '17 at 16:47

You should change your sed command to this:

sed -i '' "s#sceneFilePath:.*#sceneFilePath: \"${myfile//&/\\&}\",#" "$tmp"

This will escape each occurrence of & so that & looses it's special meaning in replacement. Unescaped & in replacement is back-reference of the full match in the substitution pattern.

score 1 · Answer 2 · answered May 09 '17 at 21:24

Just use awk:

myfile="$myfile" awk -i inplace 'match($0,/(.*sceneFilePath:).*/,a) {$0=a[1] "\"" ENVIRON["myfile"] "\""} 1' "$tmp"

That will work for ANY characters in $myfile since it's just doing a literal string operation. It uses GNU awk for the 3rd arg to match() and, less importantly, inplace editing.

score 0 · Accepted Answer · answered May 09 '17 at 16:49

Lets try something:

sed -e "s#sceneFilePath:.*,#fooFilePath: \"${myfile}\",#" index.html

results in

fooFilePath: "./video/Bentota sceneFilePath: "./video/video/960.mp4", Hikkaduwa/Hotels/River House/River House - Balapitiya.mp4",

So it matches the sceneFilePath line, replaces it, then inserts the matched content, then adds the rest of the replacement. Why?

From the sed manpage

   s/regexp/replacement/
          Attempt to match regexp against the pattern space.  If  success‐
          ful,   replace  that  portion  matched  with  replacement.   The
          replacement may contain the special character & to refer to that
          portion  of  the  pattern  space  which matched, and the special
          escapes \1 through \9 to refer  to  the  corresponding  matching
          sub-expressions in the regexp.

So Your & is not escaped and inserts the matched scheneFilePath:.*, I'd try to escape it with something like

rep=${myfile//&/\\&}
sed -i "s#sceneFilePath:.*,#sceneFilePath: \"${rep}\",#g" "${tmp}"

linux shell: sed regex working with special char

3 Answers3