635

I would like to run a find and replace on an HTML file through the command line.

My command looks something like this:

sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html > index.html

When I run this and look at the file afterward, it is empty. It deleted the contents of my file.

When I run this after restoring the file again:

sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html

The stdout is the contents of the file, and the find and replace has been executed.

Why is this happening?

codeforester
  • 39,467
  • 16
  • 112
  • 140
BBales
  • 6,588
  • 3
  • 16
  • 19
  • 14
    Perl alternative: `perl -pi -w -e 's/STRING_TO_REPLACE/REPLACE_WITH/g;' index.html` – Gjorgji Tashkovski Jun 27 '11 at 22:18
  • much related `sed` command to find a string and replace the whole line: http://stackoverflow.com/questions/11245144/replace-whole-line-containing-a-string-using-sed – cregox Jul 26 '13 at 13:14
  • See this as well: [How to make reading and writing the same file in the same pipeline always “fail”?](https://unix.stackexchange.com/a/409896/201820) on Unix & Linux SO. – codeforester Apr 04 '19 at 19:17

12 Answers12

979

When the shell sees > index.html in the command line it opens the file index.html for writing, wiping off all its previous contents.

To fix this you need to pass the -i option to sed to make the changes inline and create a backup of the original file before it does the changes in-place:

sed -i.bak s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html

Without the .bak the command will fail on some platforms, such as Mac OSX.

tripleee
  • 175,061
  • 34
  • 275
  • 318
codaddict
  • 445,704
  • 82
  • 492
  • 529
  • 24
    Saying `truncates the file` instead of `opens the file` probably makes it clearer. – Mikel Mar 02 '11 at 22:50
  • 13
    At least on my mac, the first suggestion doesn't work... if you're doing in-place replacement on a file, you have to specify an extension. You can, at least, pass in a zero-length extension though: sed -i '' s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html – Tom Lianza May 12 '11 at 03:57
  • 5
    for variables sed -i.bak 's/'$search'/'$replace'/g' index.html – Fatima Zohra Sep 19 '13 at 06:54
  • Thanks. Didn't find mentioned anywhere that the backup extension is a must in Mac OSX. Was throwing `invalid character code f` error when I tried it with a file named **foo** without providing any extension.. – SexyBeast May 01 '14 at 11:27
  • 42
    on osx, use an empty string '' as the parameter for -i, like: `sed -i '' 's/blah/xx/g'` – Pierre Houston Sep 06 '14 at 20:46
  • 4
    but what's yours `.bak` after `sed -i` ? – Patrizio Bertoni Feb 09 '15 at 09:34
  • using cygwin on Win7 64 I got the temp files and original files were gone, no replacements. – davidjmcclelland Aug 18 '15 at 19:28
  • @smallduck Thanks for your comment. I hate it creating a new bak file on mac. Your answer solves my problem. – JasmineOT Sep 15 '15 at 09:21
  • For Mac users: you can also install the gnu-sed to use instead of the BSD version used on Mac. See here: http://apple.stackexchange.com/a/195595 – Christof Jan 08 '16 at 10:51
  • 2
    By python's philosophy, "explicit is better than implicit", here goes an exact same but more readable version of the command: `sed --in-place ".bak" s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html` – Mayank Jaiswal Apr 07 '16 at 15:19
  • 1
    Just to clarify a bit further. If it had been `sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html > index_new.html` it would have worked. The confusion stems from the fact that we're reading left to right. What happens though is that the file is truncated **first** in preparation for the sed command that follows, which ends up operating on an already empty file. – Nagev Feb 16 '17 at 15:54
  • @MayankJaiswal when I copy your code I get: sed: -e expression #1, char 1: unknown command: `.' – Natan Aug 06 '22 at 15:57
220

An alternative, useful, pattern is:

sed -e 'script script' index.html > index.html.tmp && mv index.html.tmp index.html

That has much the same effect, without using the -i option, and additionally means that, if the sed script fails for some reason, the input file isn't clobbered. Further, if the edit is successful, there's no backup file left lying around. This sort of idiom can be useful in Makefiles.

Quite a lot of seds have the -i option, but not all of them; the posix sed is one which doesn't. If you're aiming for portability, therefore, it's best avoided.

Norman Gray
  • 11,978
  • 2
  • 33
  • 56
  • 10
    +1 for no backup file laying around and not clobbering input file if edit fails. Worked flawlessly on mac. – Mike Grace Jun 04 '14 at 22:20
  • 1
    Worked for me perfectly. Thank you! (on a Mac) – interested Feb 10 '16 at 14:02
  • 1
    This worked for me perfectly where on Ubuntu Server 14.04 sed -i kept zeroing out the file. – Chris Giddings Mar 21 '16 at 17:25
  • 2
    Extremely minor enhancement: `... && mv index.html{.tmp,}` – EdwardG Aug 03 '16 at 17:38
  • 5
    @EdwardGarson Indeed, that's probably what I'd use if I were typing it – I agree it's neater – but `sh` (if I recall correctly) doesn't have that `{...}` expansion. In a Makefile you might be using `sh` rather than `bash`, so if you're aiming for portability (or posixness) then you'll need to avoid that construction. – Norman Gray Aug 04 '16 at 09:05
  • Or shorter: `mv index.html $(echo index.html | sed -e 'script')` – Timo Dec 23 '17 at 10:52
99
sed -i 's/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g' index.html

This does a global in-place substitution on the file index.html. Quoting the string prevents problems with whitespace in the query and replacement.

Rich Apodaca
  • 28,316
  • 16
  • 103
  • 129
60

use sed's -i option, e.g.

sed -i bak -e s/STRING_TO_REPLACE/REPLACE_WITH/g index.html
Kevin
  • 24,871
  • 19
  • 102
  • 158
19

To change multiple files (and saving a backup of each as *.bak):

perl -p -i -e "s/\|/x/g" *

will take all files in directory and replace | with x this is called a “Perl pie” (easy as a pie)

Alex
  • 10,470
  • 8
  • 40
  • 62
Stenemo
  • 611
  • 7
  • 13
  • 2
    Good to see someone willing to look at the problem statement, and not just the tags. OP didn't specify `sed` as a requirement, only used it as the tool already tried. – Chindraba Jul 28 '18 at 23:30
14

You should try using the option -i for in-place editing.

uloBasEI
  • 4,167
  • 2
  • 19
  • 13
7

Warning: this is a dangerous method! It abuses the i/o buffers in linux and with specific options of buffering it manages to work on small files. It is an interesting curiosity. But don't use it for a real situation!

Besides the -i option of sed you can use the tee utility.

From man:

tee - read from standard input and write to standard output and files

So, the solution would be:

sed s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html | tee | tee index.html

-- here the tee is repeated to make sure that the pipeline is buffered. Then all commands in the pipeline are blocked until they get some input to work on. Each command in the pipeline starts when the upstream commands have written 1 buffer of bytes (the size is defined somewhere) to the input of the command. So the last command tee index.html, which opens the file for writing and therefore empties it, runs after the upstream pipeline has finished and the output is in the buffer within the pipeline.

Most likely the following won't work:

sed s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html | tee index.html

-- it will run both commands of the pipeline at the same time without any blocking. (Without blocking the pipeline should pass the bytes line by line instead of buffer by buffer. Same as when you run cat | sed s/bar/GGG/. Without blocking it's more interactive and usually pipelines of just 2 commands run without buffering and blocking. Longer pipelines are buffered.) The tee index.html will open the file for writing and it will be emptied. However, if you turn the buffering always on, the second version will work too.

xealits
  • 4,224
  • 4
  • 27
  • 36
  • 3
    The output file of tee is also opened immediately resulting in an empty index.html for the entire command. – sjngm Feb 22 '16 at 11:32
  • 4
    **This will _corrupt_ any input file that is larger than the _pipeline buffer_ (which is typically 64KB)**. (@sjngm: the file isn't truncated instantly as with `>`, but the point stands that it's a broken solution that is likely to result in data loss). – mklement0 Jul 03 '17 at 14:51
6
sed -i.bak "s#https.*\.com#$pub_url#g" MyHTMLFile.html

If you have a link to be added, try this. Search for the URL as above (starting with https and ending with.com here) and replace it with a URL string. I have used a variable $pub_url here. s here means search and g means global replacement.

It works !

Kaey
  • 4,615
  • 1
  • 14
  • 18
5

The problem with the command

sed 'code' file > file

is that file is truncated by the shell before sed actually gets to process it. As a result, you get an empty file.

The sed way to do this is to use -i to edit in place, as other answers suggested. However, this is not always what you want. -i will create a temporary file that will then be used to replace the original file. This is problematic if your original file was a link (the link will be replaced by a regular file). If you need to preserve links, you can use a temporary variable to store the output of sed before writing it back to the file, like this:

tmp=$(sed 'code' file); echo -n "$tmp" > file

Better yet, use printf instead of echo since echo is likely to process \\ as \ in some shells (e.g. dash):

tmp=$(sed 'code' file); printf "%s" "$tmp" > file
Andrzej Pronobis
  • 33,828
  • 17
  • 76
  • 92
  • 2
    +1 for preserving links. It also works with a temporary file: `sed 'code' file > file.tmp; cat file.tmp > file; rm file.tmp` – dashohoxha Nov 26 '17 at 23:49
3

And the ed answer: 

printf "%s\n" '1,$s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g' w q | ed index.html

To reiterate what codaddict answered, the shell handles the redirection first, wiping out the "input.html" file, and then the shell invokes the "sed" command passing it a now empty file.

Community
  • 1
  • 1
glenn jackman
  • 238,783
  • 38
  • 220
  • 352
  • 2
    quick question, why people keep giving "the `ed` version" of `sed` answers? does it perform faster? – cregox Jul 26 '13 at 12:58
  • 6
    Some `sed`s don't implement `-i` to edit in-place. `ed` is ubiquitous and does let you save your edits to the original file. Plus it's always good to have lots of tools in your kit. – glenn jackman Jul 26 '13 at 13:48
  • ok, cool. so, performance wise, they're the same I suppose. thanks! – cregox Jul 26 '13 at 13:51
  • Hi this is bit late I know but I cant pass any variable in this code. like, `printf "%s\n" '1,$s/^STRING_TO_REPLACE.*/$MODPATH/g' w q | ed $SERVICESH > /dev/null 2>&1`. I want to pass $MODPATH as replace string but can't make it working – Some53 Apr 22 '21 at 17:59
0

I was searching for the option where I can define the line range and found the answer. For example I want to change host1 to host2 from line 36-57. 

sed '36,57 s/host1/host2/g' myfile.txt > myfile1.txt

You can use gi option as well to ignore the character case. 

sed '30,40 s/version/story/gi' myfile.txt > myfile1.txt
-1

With all due respect to the above correct answers, it's always a good idea to "dry run" scripts like that, so that you don't corrupt your file and have to start again from scratch.

Just get your script to spill the output to the command line instead of writing it to the file, for example, like that:

sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html

OR

less index.html | sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g

This way you can see and check the output of the command without getting your file truncated.

Nestor Milyaev
  • 5,845
  • 2
  • 35
  • 51