6

I am trying to write a shell script that will replace whatever characters/strings I choose using sed. My first attempt worked with the exception of special characters. I have been trying to use sed to fix the special characters so that they too will be searched for or replaced. I decided to simplify the script for testing purposed, and just deal with a single offending character. However, I am still having problems.

Edited Script

#! /bin/sh
oldString=$1
newString=$2
file=$3

oldStringFixed=$(echo "$oldString" | sed 's/\\/\\\\/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\[/\\\[/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\]/\\\]/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\^/\\\^/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\*/\\\*/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\+/\\\+/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\./\\\./g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\$/\\\$/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\-/\\\-/g')

sed -e "s/$oldStringFixed/$newString/g" "$file" > newfile.updated
mv newfile.updated "$file"#! /bin/sh

In case it is not clear, I am trying to search through oldString for the [ character, and replace it with an escaped version and assign the results to oldStringFixed (do I need the backticks for this?). The bottom two lines are slightly modified versions of my original script that I believe works correctly.

When I echo the fixed string, nothing is displayed, and sed outputs an error

sed: can't read [: No such file or directory

Can anyone explain what Is wrong with my first sed line?

EDIT:

Thanks to Jite, the script is working better. However, I am still having a problem with replacing single quoted characters with spaces, i.e. ' *'. The new version is above.

user3074091
  • 71
  • 1
  • 2
  • 5
  • there is lot more specias char than "[" like . \ * + ? for OldString and \ & / for newString – NeronLeVelu Dec 06 '13 at 11:51
  • Pretty similar to [this](http://stackoverflow.com/questions/20427289/quoting-special-characters-with-sed/20427706) question. – devnull Dec 06 '13 at 17:07

5 Answers5

3

I suggest two improvements:

  1. Do not stack calls to sed as you do, instead pack all of them in a single function, as escape_string below.

  2. You can use a fancy delimiter for the sed substitute command to avoid issues linked to / being part of the strings involved.

With these changes, your script looks like: 

#! /bin/sh
oldString="$1"
newString="$2"
file="$3"

escape_string()
{
   printf '%s' "$1" | sed -e 's/[][\\^*+.$-]/\\\1/g'
}

fancyDelim=$(printf '\001')

oldStringFixed=$(escape_string "$oldString")
sed -e "s$fancyDelim$oldStringFixed$fancyDelim$newString${fancyDelim}g" "$file" \
  > newfile.updated
mv newfile.updated "$file"
tripleee
  • 175,061
  • 34
  • 275
  • 318
The Show that never ends
  • 160
  • 3
  • 10
  • 1
    You can easily generalize to `s/[][\\^*+.$-]/\\\1/g` (and I'm not sure why you want to include the hyphen, so I suggest you take it out). – tripleee Sep 13 '14 at 09:36
  • Great, I added your changes! – The Show that never ends Sep 13 '14 at 10:41
  • 2
    This doesn't work for me, I get "sed: -e expression #1, char 21: invalid reference \1 on `s' command's RHS" – Ben Farmer May 09 '17 at 05:51
  • @BenFarmer This does not work when there is no special characters in the string. If you have a regular string that includes no special characters, there will be no value for \1 (which is match 1) so this will fail. You cannot use enumerated matches if the values may not contain a match. – Kim Gentes Dec 01 '21 at 21:09
3

To replace values containing special characters try using sed with "|" instead of "/"

Eg: sed -i 's|'$original_value'|'$new_value'|g'

where original_value="comprising_special_char_/" new_value="comprising_new_special_char:"

Sushant Dhingra
  • 142
  • 1
  • 4
2

Change:

oldStringFixed= `sed 's/\[/\[/g' "$oldString"\`

to:

oldStringFixed=$(echo "$oldString" | sed 's/\[/\\\[/g')

Problem 1: Space after =, it's not allowed when assigning shell variables.

Problem 2: sed expects a file as input, not a string. You may pipe it as my solution does though.

Problem 3: You need to escape the backslash first \\, then you need to escape your char \[, totalling \\\[ :)

Side note: I changed `` to $() since the latter is the recommended praxis (due to nesting, another topic).

Jite
  • 4,250
  • 1
  • 17
  • 18
  • Thanks for the help. Your changes work great, and I have added additional sed statements to deal with other problem characters. However, when I try to replace a single quoted character with a space such as ' *', the output is incorrect. Is there anyway this can be fixed? – user3074091 Dec 06 '13 at 16:58
  • It can all be fixed. Frankly, for this case, I'd write a file `sed.script` that contains all the commands to be executed, one per line, and then use `sed -f sed.script` just once, rather than reinvoking `sed` many times over. This also means you don't have to escape the script against the shell — it makes life easier. – Jonathan Leffler Dec 06 '13 at 17:06
  • @user3074091 I don't get your question. Please update your question with an example input and what output you expect, hopefully we can come up with an answer. – Jite Dec 09 '13 at 13:20
0

For me it was just a nightmare trying to get sed to do this for the general case. I gave up and wrote a short Python code to replace sed:

#!/usr/bin/python
# replace.py
import sys

# Replace string in a file (in place)
match=sys.argv[1]
replace=sys.argv[2]
filename=sys.argv[3]

print "Replacing strings in",filename

with open(filename,"r") as f:
  data = f.read().replace(match,replace)

with open(filename,"w") as f:
  f.write(data)

Which can then be used like:

#!/bin/bash
orig='<somethinghorrible>'
out='<replacement>'
python replace.py "$orig" "$out" myfile.txt
Ben Farmer
  • 2,387
  • 1
  • 25
  • 44
0

you can use this for replacing " with \" sed 's/\"/\\\"/g' filename

abrarcv170
  • 27
  • 7