How do you grep through code that lives in many different directories?

Question

I'm working on a Python program that makes heavy use of eggs (Plone). That means there are 198 directories full of Python code I might want to search through while debugging. Is there a good way to search only the .py files in only those directories, avoiding unrelated code and large binary files?

score 30 · Accepted Answer · edited Jun 20 '20 at 09:12

30

find DIRECTORY -name "*.py" | xargs grep PATTERN

By the way, since writing this, I have discovered ack, which is a much better solution.

(And since that edit, I have discovered ag).

edited Jun 20 '20 at 09:12

Community

1
1

answered May 14 '09 at 20:00

Steve B.

55,454
12
93
132

find `echo $PYTHONPATH | sed "s/:/ /g"` -name "*.py" | xargs grep PATTERN – joeforker May 14 '09 at 20:07
that's nice. Hadn't thought of that. – Steve B. May 14 '09 at 20:14
fortunately none of the entries in $PYTHONPATH contain a ' '. – joeforker May 14 '09 at 20:32
3

find DIRECTORY -name '*.py' -exec grep PATTERN {} + – Marius Gedminas Sep 17 '09 at 20:54
In all these examples, in place of DIRECTORY you can have more than one directory if necessary (eg: find /usr/local /opt -name ...) – Bryan Oakley Sep 17 '09 at 21:09
find DIRECTORY -name "*.py" -print | xargs grep -iRn --color PATTERN – Nanda Kishore Oct 21 '09 at 12:26
the "xargs grep" will first find all files and then grep them. the "exec grep" will not display the matching file name as there is only one argument every time. => "find... | xargs -n2 grep PATTERN" It will start grepping as soon as it finds two files and it will show the filename. – Eric Darchis Jul 14 '10 at 16:02

score 26 · Answer 2 · answered May 14 '09 at 20:04

26

grep -r -n "PATTERN" --include="*.py" DIRECTORY

answered May 14 '09 at 20:04

monowerker

2,951
1
25
23

I like the -n to include line numbers. Surprisingly, the `find | xargs` version is slightly faster. – joeforker May 14 '09 at 20:27
I find that grep tends to speed up in successive searches though. – Dana the Sane May 14 '09 at 21:43
Should may be considered as the right answer since the question is only about grep, not find + grep – Quentin Dec 14 '18 at 10:09

Brian Agnew · Answer 3 · 2011-02-26T14:16:41.457

19

I would strongly recommend ack, a grep substitute, "aimed at programmers with large trees of heterogeneous source code" (from the website)

edited Feb 26 '11 at 14:16

answered May 14 '09 at 20:15

Brian Agnew

268,207
37
334
440

score 7 · Answer 4 · answered Jun 16 '09 at 14:43

I also use ack a lot these days. I did tweak it a bit to find all the relevant file types:

# Add zcml to the xml type:
--type-add
xml=.zcml

# Add more files the plone type:
--type-add
plone=.dtml,.zpt,.kss,.vpy,.props

# buildout config files
--type-set
buildout=.cfg

# Include our page templates to the html type so we can limit our search:
--type-add
html=.pt,.zpt

# Create txt file type:
--type-set
txt=.txt,.rst

# Define i18n file types:
--type-set
i18n=.pot,.po

# More options
--follow
--ignore-case
--nogroup

Important to remember is that ack won't find files if the extension isn't in its configuration. See "ack --help-types" for all the available types.

I also assume you are using omelette so you can grep/ack/find all the related files?

score 4 · Answer 5 · answered Feb 27 '11 at 05:00

4

This problem was the motivation for the creation of collective.recipe.omelette. It is a buildout recipe which can symlink all the eggs from your working set into one directory structure, which you can point your favorite search utility at.

answered Feb 27 '11 at 05:00

David Glick

5,422
17
23

The grep-oriented answers are red herrings. They'll lead you to find multiple versions of files, all but one in un-used versions of the code (buildout may have fetched different egg releases over time). Use omelette, and grep the symlink structure which it generates. – Jean Jordaan Mar 14 '11 at 10:12

score 4 · Answer 6 · edited Jul 12 '18 at 02:27

4

find <directory> -name '*.py' -exec grep <pattern> {} \;

edited Jul 12 '18 at 02:27

phuclv

37,963
15
156
475

answered May 14 '09 at 20:04

Mike

412
3
7

1

This version is 26 times slower than the | xargs or standalone grep solution because it executes grep 16,836 times instead of once. – joeforker May 14 '09 at 20:31
3

but if you end it with a + instead of \;, then it's equivalent to the xargs solution, except doesn't break if your pathnames have spaces in them. – Marius Gedminas Sep 17 '09 at 20:56

score 2 · Answer 7 · answered Sep 17 '09 at 20:59

There's also GNU idutils if you want to grep for identifiers in a large source tree very very quickly. It requires building a search database in advance, by running mkid (and tweaking its config file to not ignore .py files). z3c.recipe.tag takes care of that, if you use buildout.

score 2 · Answer 8 · answered Feb 27 '11 at 04:21

Just in case you want a non-commandline OSS solution...

I use pycharm. It has built in support for buildout. You point it at a buildout generated bin/instance and it sets the projects external dependencies to all the eggs used by the instance. Then all the IDE's introspection and code navigation work nicely. Goto definition, goto instances, refactoring support and of course search.

score 2 · Answer 9 · answered Feb 28 '11 at 09:19

2

I recomend grin to search, omelette when working with plone and the pydev-feature 'Globals browser' (with eclipse or aptana studio).

answered Feb 28 '11 at 09:19

pbauer

635
5
8

Here are helpful scripts to import omelette + src folder to Eclipse: http://svn.plone.org/svn/collective/collective.eclipsescripts/trunk/README.txt – Mikko Ohtamaa Mar 25 '11 at 15:15

score 2 · Answer 10 · answered Mar 04 '11 at 03:56

2

And simply because there are not enough answers...

If you're developing routinely, it's well worth the effort to install Eclipse with Pydev (or even easier, Aptana Studio - which is a modified Eclipse), in which case the find tools are right there.

answered Mar 04 '11 at 03:56

Auspex

2,175
15
35

Here is an a script which allows import buildout + omelette to Aptana: http://svn.plone.org/svn/collective/collective.eclipsescripts/trunk/README.txt – Mikko Ohtamaa Mar 25 '11 at 15:15

score 2 · Answer 11 · edited Nov 09 '20 at 09:05

2

OpenGrok is an excellent choice for source searching and navigation. Runs on Java, though.

I really wish there was something like https://oracle.github.io/opengrok/

edited Nov 09 '20 at 09:05

Suncatcher

10,355
10
52
90

answered Mar 07 '11 at 21:01

Davi Lima

800
1
6
20

Dan Jacka · Answer 12 · 2011-02-28T20:27:54.710

My grepping life is way more satisfying since discovering Emacs' rgrep command.

Say I want to find 'IPortletDataProvider' in Plone's source. I do:

M-x rgrep
Emacs prompts for the search string (IPortletDataProvider)
... then which files to search (*.py)
... then which directory (~/Plone/buildout-cache/eggs). If I'm already editing a file, this defaults to that file's directory, which is usually exactly what I want.

The results appear in a new buffer. At the top is the find | xargs grep command Emacs ran. All matches are highlighted. I can search the buffer using the standard text search commands. Best of all, I can hit Enter (or click) on a match to open that file.

It's a pretty nice way to work. I like that I don't have to remember find | xargs grep argument sequences, but that all that power is there if I need it.

How do you grep through code that lives in many different directories?

12 Answers12