Environment variable substitution in sed

Question

If I run these commands from a script:

#my.sh
PWD=bla
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
xxx
bla

it is fine.

But, if I run:

#my.sh
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
$ sed: -e expression #1, char 8: Unknown option to `s' 

I read in tutorials that to substitute environment variables from shell you need to stop, and 'out quote' the $varname part so that it is not substituted directly, which is what I did, and which works only if the variable is defined immediately before.

How can I get sed to recognize a $var as an environment variable as it is defined in the shell?

Answer 1

Your two examples look identical, which makes problems hard to diagnose. Potential problems:

1. You may need double quotes, as in sed 's/xxx/'"$PWD"'/'

2. $PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.

To nail both issues at once, perhaps

sed 's@xxx@'"$PWD"'@'

Answer 2

In addition to Norman Ramsey's answer, I'd like to add that you can double-quote the entire string (which may make the statement more readable and less error prone).

So if you want to search for 'foo' and replace it with the content of $BAR, you can enclose the sed command in double-quotes.

sed 's/foo/$BAR/g'
sed "s/foo/$BAR/g"

In the first, $BAR will not expand correctly while in the second $BAR will expand correctly.

Answer 3

Another easy alternative:

Since $PWD will usually contain a slash /, use | instead of / for the sed statement:

sed -e "s|xxx|$PWD|"

Answer 4

You can use other characters besides "/" in substitution:

sed "s#$1#$2#g" -i FILE

Answer 5

一. bad way: change delimiter

sed 's/xxx/'"$PWD"'/'
sed 's:xxx:'"$PWD"':'
sed 's@xxx@'"$PWD"'@'

maybe those not the final answer,

you can not known what character will occur in $PWD, / : OR @.
if delimiter char in $PWD, they will break the expression

the good way is replace(escape) the special character in $PWD.

二. good way: escape delimiter

for example: try to replace URL as $url (has : / in content)

x.com:80/aa/bb/aa.js

in string $tmp

<a href="URL">URL</a>

A. use / as delimiter

escape / as \/ in var (before use in sed expression)

## step 1: try escape
echo ${url//\//\\/}
x.com:80\/aa\/bb\/aa.js   #escape fine

echo ${url//\//\/}
x.com:80/aa/bb/aa.js      #escape not success

echo "${url//\//\/}"
x.com:80\/aa\/bb\/aa.js   #escape fine, notice `"`


## step 2: do sed
echo $tmp | sed "s/URL/${url//\//\\/}/"
<a href="x.com:80/aa/bb/aa.js">URL</a>

echo $tmp | sed "s/URL/${url//\//\/}/"
<a href="x.com:80/aa/bb/aa.js">URL</a>

OR

B. use : as delimiter (more readable than /)

escape : as \: in var (before use in sed expression)

## step 1: try escape
echo ${url//:/\:}
x.com:80/aa/bb/aa.js     #escape not success

echo "${url//:/\:}"
x.com\:80/aa/bb/aa.js    #escape fine, notice `"`


## step 2: do sed
echo $tmp | sed "s:URL:${url//:/\:}:g"
<a href="x.com:80/aa/bb/aa.js">x.com:80/aa/bb/aa.js</a>

Answer 6

With your question edit, I see your problem. Let's say the current directory is /home/yourname ... in this case, your command below:

sed 's/xxx/'$PWD'/'

will be expanded to

sed `s/xxx//home/yourname//

which is not valid. You need to put a \ character in front of each / in your $PWD if you want to do this.

Answer 7

Actually, the simplest thing (in GNU sed, at least) is to use a different separator for the sed substitution (s) command. So, instead of s/pattern/'$mypath'/ being expanded to s/pattern//my/path/, which will of course confuse the s command, use s!pattern!'$mypath'!, which will be expanded to s!pattern!/my/path!. I’ve used the bang (!) character (or use anything you like) which avoids the usual, but-by-no-means-your-only-choice forward slash as the separator.

Answer 8

VAR=8675309
echo "abcde:jhdfj$jhbsfiy/.hghi$jh:12345:dgve::" |\
sed 's/:[0-9]*:/:'$VAR':/1' 

where VAR contains what you want to replace the field with

Answer 9

Dealing with VARIABLES within sed

[root@gislab00207 ldom]# echo domainname: None > /tmp/1.txt

[root@gislab00207 ldom]# cat /tmp/1.txt

domainname: None

[root@gislab00207 ldom]# echo ${DOMAIN_NAME}

dcsw-79-98vm.us.oracle.com

[root@gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: ${DOMAIN_NAME}/g'

 --- Below is the result -- very funny.

domainname: ${DOMAIN_NAME}

 --- You need to single quote your variable like this ... 

[root@gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: '${DOMAIN_NAME}'/g'


--- The right result is below 

domainname: dcsw-79-98vm.us.oracle.com

Answer 10

If your replacement string may contain other sed control characters, then a two-step substitution (first escaping the replacement string) may be what you want:

PWD='/a\1&b$_' # these are problematic for sed
PWD_ESC=$(printf '%s\n' "$PWD" | sed -e 's/[\/&]/\\&/g')
echo 'xxx' | sed "s/xxx/$PWD_ESC/" # now this works as expected

Answer 11

I had similar problem, I had a list and I have to build a SQL script based on template (that contained @INPUT@ as element to replace):

for i in LIST 
do
    awk "sub(/\@INPUT\@/,\"${i}\");" template.sql >> output
done

Answer 12

for me to replace some text against the value of an environment variable in a file with sed works only with quota as the following:

sed -i 's/original_value/'"$MY_ENVIRNONMENT_VARIABLE"'/g' myfile.txt

BUT when the value of MY_ENVIRONMENT_VARIABLE contains a URL (ie https://andreas.gr) then the above was not working. THEN use different delimiter:

sed -i "s|original_value|$MY_ENVIRNONMENT_VARIABLE|g" myfile.txt