5

I have read all the similar questions on this topic, but didn't find a matching question to what I am experiencing. I apologise if this has been answered already.

Inside a bash script I wrote, there is a very simple sed command, which does not seem to be working. There are no errors, and the command works perfectly when run from the command line.

In the output from set -x I can see the sed command executing perfectly.

GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)

Bash script: (toned down for easier comprehension)

#!/bin/bash -x

# This script has the exact same sed command as used on cli

contact='"tim@email.com"'

sed -i "/$contact/d" /home/tim/Desktop/file.txt

exit

Shell output:

tim@ubuntu:~/Desktop$ cat file.txt
t,b,tim@email.com
tim@ubuntu:~/Desktop$ ./test.sh 
+ contact='"tim@email.com"'
+ sed -i '/"tim@email.com"/d' /home/tim/Desktop/file.txt
+ exit
tim@ubuntu:~/Desktop$ cat file.txt
t,b,tim@email.com
tim@ubuntu:~/Desktop$ sed -i "/"tim@email.com"/d" /home/tim/Desktop/file.txt
tim@ubuntu:~/Desktop$ cat file.txt
tim@ubuntu:~/Desktop$

I assume I am missing something very obvious, but I am done staring at it hoping for the answer to jump off the screen and slap me in the face. Please help :-)

Tim

sjsam
  • 21,411
  • 5
  • 55
  • 102
asimovwasright
  • 838
  • 1
  • 11
  • 28
  • Why do you have 2 quotes in `contact='"tim@email.com"'`. It should just be: `contact='tim@email.com'` – anubhava Mar 19 '15 at 10:28
  • Your commands are _not_ identical: inside the script you enclose the match pattern with single quotes, on the cli with doublequotes. That means, sed called from your bash script matches against the email address enclosed in double quotes, which it isn't in your file. – collapsar Mar 19 '15 at 10:32
  • @anubhava Sorry, I should I guessed someone would ask that. In the dumbed down script it seems pointless, but in the actual script there is a base64 decoded string, which is then openssl decrypted and becomes the $contact variable, and is quoted. I would rather not try to remove the quotes since the variable is used in many other functions which would then have to be rewritten. Either way, it works on the command line with the quotes. – asimovwasright Mar 19 '15 at 10:33
  • @collapsar That's what I was thinking, except in the script, there are no single quotes, this seems to happen only in the set -x output. – asimovwasright Mar 19 '15 at 10:34
  • it works on the command line with the single quotes ? – collapsar Mar 19 '15 at 10:34
  • in the script your variable contains the double quotes around the mail address- these chars are missing in your file. that's why it won't match. – collapsar Mar 19 '15 at 10:36
  • No, but how to stop the script from adding the single quotes. – asimovwasright Mar 19 '15 at 10:36
  • The double quotes are the problem: you are trying to match `"tim@email.com"` while your file only contains `tim@email.com`. – collapsar Mar 19 '15 at 10:37
  • Yes, I know this, but **how** can I avoid the single quoting on the script - look at the script, there are no single quotes. – asimovwasright Mar 19 '15 at 10:37
  • Please read the comments: the *double* quotes are the problem - they are part of the `$contact` variable! – collapsar Mar 19 '15 at 10:38
  • I just can't change the variable from having quotes, and I can't change the file from not having them. – asimovwasright Mar 19 '15 at 10:39
  • Execute `sed -i '/"tim@email.com"/d' /home/tim/Desktop/file.txt` on the command line. – collapsar Mar 19 '15 at 10:39
  • If the sed command ran the way it was scripted, the quotes would not matter. That is my question, how can I get sed to run as requested, without the single quotes – asimovwasright Mar 19 '15 at 10:39

2 Answers2

9

There are double quotes around the mail address in the $contact script variable that are missing from the command line call:

# case 1 - works
# only the sed pattern delimiters are enclosed in quotes and these quotes will be stripped by the shell. 
sed -i "/"tim@email.com"/d" ./file.txt; cat file.txt

# case 2 - fails
# escaping with \ turns dquotes #2,3 from shell-level delimiters to char literals w/o special  semantics.
sed -i "/\"tim@email.com\"/d" ./file.txt; cat file.txt

# case 3 - fails
# Single quotes enclose the complete sed pattern spec which comprises double quotes enclosing the mail address
sed -i '/"tim@email.com"/d' ./file.txt; cat file.txt

# case 4 - works
sed -i "/tim@email.com/d" ./file.txt; cat file.txt

# case 5 - works
sed -i '/tim@email.com/d' ./file.txt; cat file.txt

This would explain the different behavior of the script vs. the cli call.

The OP pointed out that he needs the double quotes in the real script. that may be so, however, if these doublequotes aren't present in the file, there will be no match.

A solution would be to preprocess the file ( if necessary, work on a copy ) with sed:

sed -i 's/,/","/g; s/^/"/; s/$/"/' ./file.txt

This command assumes a comma-separated list of items on each line with no item containing double quotes. It will wrap each item in double quotes so they will match against the search pattern in the original script's $contact variable.

Alternative (adapted from this SO answer [that I have not been the author of])

Another option is changing the relevant portion of the script be deriving a second variable from $contact:

contact='"tim@email.com"'
c2=$(echo $contact | tr -d '"')

sed -i "/$c2/d"  /home/tim/Desktop/file.txt
Community
  • 1
  • 1
collapsar
  • 17,010
  • 4
  • 35
  • 61
  • Why are the double quotes stripped by the shell and replaced with single quotes? If I could understand that, I would be very happy :-) – asimovwasright Mar 19 '15 at 11:44
  • 1
    The quoting is a generic mechanism to specify string literals containing characters with special semantics to the command processor - eg. whitespace. Double and single quotes differ (as a rule of thumb) in that double quotes still allow for variable expansion. So in any case, sed does not get to see the outermost pair of quotes. In your command line call, there are _2_ outermost pairs and effectively 3 adjacent strings that are implicitly wrapped into a single argument to sed (the pattern expression). I've added case 2, contrast that with cases 1,3. – collapsar Mar 19 '15 at 12:04
  • Right, got it, and thank you for such a clear explanation! I think I will have to use your prepocessor to alter the match criteria. Very helpful :-) – asimovwasright Mar 19 '15 at 12:09
3

This is just an addition to collapsar's answer which already solved the issue.

When we use sed in bash script, bash script acts as a wrapper for sed. This has two purposes

  • sed command can be executed as if it is executed outside the bash.

  • bash script wrapper helps sed to communicate with the outside world, using environment variables.

For example, suppose that the file testfile contains two lines

somebody@email.com
tim@email.com

Now If I would like to write a bash script which helps sed replace the lines that contain tim@email.com my script sedscript would be like this :

#!/bin/bash
contact='tim@email.com'
sed -i "/$contact/d" $1

Now I would execute the script like below

./sedscript testfile

to remove all the lines containing tim@email.com.

In fact you can replace $1 with the actual file name. But the important point to note, as mentioned in the previous answer, is that whenever we use a bash variable inside the sed command, always enclose the command in double quotes. Only then bash would replace the variable with corresponding string before passing it to sed.

sjsam
  • 21,411
  • 5
  • 55
  • 102