Using grep and sed to find and replace a string

Question

I am using the following to search a directory recursively for specific string and replace it with another:

grep -rl oldstr path | xargs sed -i 's/oldstr/newstr/g'

This works okay. The only problem is that if the string doesn't exist then sed fails because it doesn't get any arguments. This is a problem for me since i'm running this automatically with ANT and the build fails since sed fails.

Is there a way to make it fail-proof in case the string is not found?

I'm interested in a one line simple solution I can use (not necessarily with grep or sed but with common unix commands like these).

the reason i want to keep it as simple as possible is because my script connects to a remote server and runs this line with SSH. if i would use a shell script for this, i would have to copy the shell script to the server before and then run it there. i'm trying to avoid it and keep it simple. thanks. — Michael, May 30 '11 at 16:14
http://superuser.com/questions/257250/how-to-run-sed-on-over-10-million-files-in-a-directory — , May 30 '11 at 16:47
Coming here from Google, I was looking for the exact string given in the question! (I didn't have the problem the OP was suffering from) — LondonRob, Nov 17 '15 at 12:34

score 78 · Accepted Answer · answered May 30 '11 at 16:13

78

You can use find and -exec directly into sed rather than first locating oldstr with grep. It's maybe a bit less efficient, but that might not be important. This way, the sed replacement is executed over all files listed by find, but if oldstr isn't there it obviously won't operate on it.

find /path -type f -exec sed -i 's/oldstr/newstr/g' {} \;

answered May 30 '11 at 16:13

Michael Berkowski

267,341
46
444
390

1

@Vlad: It is, because you're running a separate `sed` for each file instead of letting `xargs` batch them. That said, unless you're talking about several thousand tiny files, you're unlikely to notice a difference. – geekosaur May 30 '11 at 16:29
@geekosaur: Oh, right.. I haven't thought about a number of execs shell should make. Good point! – May 30 '11 at 16:33
thanks, i tried your solution. it did solve my return value problem, now it returns 0 when it doesn't find anything. the problem is that it returns : `sed: no input files` even though i have files with the oldstring in the directory. know why ? – Michael May 30 '11 at 16:33
i'm using this in directory tree with possibly thousand of files to look in (although the amount of files that actually need change is not big) is that an issue ? – Michael May 30 '11 at 16:35
2

My problem with this: the timestamp of every file will get reset to "right now" as if they were all "touched", even if no edits occurred in the file. – pbarill Apr 14 '13 at 03:33
1

I've seen this "Hello world" example of using find all over the place, but nothing that constrains it from running sed on every file, including binaries. I have yet to find a working example with pipes, and I've tried a number of ways to hack the simplistic version of very limited utility. – Jerry Miller Jun 17 '16 at 12:57
1

@JerryMiller you can of course limit `find` to specific filename patterns, which is what I often do: `find -name "*.c" -o -name "*.h" -type f -exec sed...` would modify all the .c and .h files for example... To exclude binary files, it can get more complicated: http://unix.stackexchange.com/questions/46276/finding-all-non-binary-files – Michael Berkowski Jun 17 '16 at 13:42
But if you want a solution with `grep` and `sed` : `find . -name "find_files" -exec grep -l "text_to_find" {} \; -exec sed -i 's/change_text/to_this_text/g' {} \;` The `grep` `-l` option returns a list of files, instead of the text found. – trogne Nov 21 '16 at 19:20
Worked for me. Made me save quite much time. Thanks! – oz19 Feb 25 '21 at 11:41

score 18 · Answer 2 · answered May 30 '11 at 20:04

18

Your solution is ok. only try it in this way:

files=$(grep -rl oldstr path) && echo $files | xargs sed....

so execute the xargs only when grep return 0, e.g. when found the string in some files.

answered May 30 '11 at 20:04

clt60

62,119
17
107
194

2

That is a nice underestimated solution! This is what I needed to execute sed over a list of files output of grep, and avoid the error: "can't read .... No such file or directory". Thanks for your answer! – Leopoldo Sanczyk Dec 24 '14 at 13:51
3

Agree with @Leopoldo. This looks like the most Posixy and portable (for OS like Solaris) and the one with the least side effects (not touching every file mtime, etc). `files=$(grep -rl oldstr path | cut -f 1 -d ':' | sort | uniq)` should build a smaller list where the files are listed once. – jww Oct 25 '17 at 00:28

Michael · Answer 3 · 2014-01-28T00:56:32.680

9

I have taken Vlad's idea and changed it a little bit. Instead of

grep -rl oldstr path | xargs sed -i 's/oldstr/newstr/g' /dev/null

Which yields

sed: couldn't edit /dev/null: not a regular file

I'm doing in 3 different connections to the remote server

touch deleteme
grep -rl oldstr path | xargs sed -i 's/oldstr/newstr/g' ./deleteme
rm deleteme

Although this is less elegant and requires 2 more connections to the server (maybe there's a way to do it all in one line) it does the job efficiently as well

edited Jan 28 '14 at 00:56

answered May 30 '11 at 17:14

Michael

22,196
33
132
187

And this script save my life! – workdreamer Oct 09 '13 at 15:24

score 8 · Answer 4 · answered May 30 '11 at 16:17

8

Standard xargs has no good way to do it; you're better off using find -exec as someone else suggested, or wrap the sed in a script which does nothing if there are no arguments. GNU xargs has the --no-run-if-empty option, and BSD / OS X xargs has the -L option which looks like it should do something similar.

answered May 30 '11 at 16:17

geekosaur

59,309
11
123
114

thanks, i tried to use --no-run-if-empty but it still returns nonzero code (returns 1) and that would also trigger a build fail for me. how generic and common is `find` command ? – Michael May 30 '11 at 16:25
`find -exec` goes back to 7th Research Edition UNIX; it should work anywhere that has `find` installed. – geekosaur May 30 '11 at 16:27
1

+1 I'm surprised the `xargs -r` fix is not mentioned in more answers. For this limited use case, it's simple and sufficient. – tripleee Aug 30 '13 at 10:48

score 4 · Answer 5 · answered May 30 '11 at 16:37

4

I think that without using -exec you can simply provide /dev/null as at least one argument in case nothing is found:

grep -rl oldstr path | xargs sed -i 's/oldstr/newstr/g' /dev/null

answered May 30 '11 at 16:37

2

could be a very neat workaround. sadly i get `sed: couldn't edit /dev/null: not a regular file` :) – Michael May 30 '11 at 16:41
@michael: Turns out - running anything like this will actually crash the shell because of too many arguments... See http://superuser.com/questions/257250/how-to-run-sed-on-over-10-million-files-in-a-directory – May 30 '11 at 16:48
there are a lot of files in my directory but only few of them contain the string i want to change. so i am not so worried about too many arguments to sed. – Michael May 30 '11 at 17:04
@michael: The solution listed there with 10 votes should do the job. – May 30 '11 at 17:08

score 1 · Answer 6 · answered May 22 '13 at 11:47

My use case was I wanted to replace foo:/Drive_Letter with foo:/bar/baz/xyz In my case I was able to do it with the following code. I was in the same directory location where there were bulk of files.

find . -name "*.library" -print0 | xargs -0 sed -i '' -e 's/foo:\/Drive_Letter:/foo:\/bar\/baz\/xyz/g'

hope that helped.

score 0 · Answer 7 · answered Oct 27 '20 at 04:06

Not sure if this will be helpful but you can use this with a remote server like the example below

ssh example.server.com "find /DIR_NAME -type f -name "FILES_LOOKING_FOR" -exec sed -i 's/LOOKINGFOR/withThisString/g' {} ;"

replace the example.server.com with your server replace DIR_NAME with your directory/file locations replace FILES_LOOKING_FOR with files you are looking for replace LOOKINGFOR with what you are looking for replace withThisString with what your want to be replaced in the file

score -1 · Answer 8 · edited Jan 29 '16 at 11:12

If you are to replace a fixed string or some pattern, I would also like to add the bash builtin pattern string replacement variable substitution construct. Instead of describing it myself, I am quoting the section from the bash manual:

${parameter/pattern/string}

The pattern is expanded to produce a pattern just as in pathname expansion. parameter is expanded and the longest match of pattern against its value is replaced with string. If pattern begins with /, all matches of pattern are replaced with string. Normally only the first match is replaced. If pattern begins with #, it must match at the beginning of the expanded value of parameter. If pattern begins with %, it must match at the end of the expanded value of parameter. If string is null, matches of pattern are deleted and the / following pattern may be omitted. If parameter is @ or *, the substitution operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with @ or *, the substitution operation is applied to each member of the array in turn, and the expansion is the resultant list.

Using grep and sed to find and replace a string

8 Answers8

Linked