how to find a search term in source code

Question

I'm looking for a way to search for a given term in a project's C/C++ code, while ignoring any occurrences in comments and strings.

As the code base is rather large, i am searching for a way to automatically identify the lines of code matching my search term, as they need manual inspection.

If possible I'd like to perform the search on my linux system.

background

the code base in question is a realtime signal processing engine with a large number of 3rd party plugins. plugins are implemented in a variety of languages (mostly C, but also C++ and others; currently I only care for those two), no standards have been enforced.

our code base currently uses the built-in type float for floating-point numbers and we would like to replace that with a typedef that would allow us to use doubles. we would like to find all occurrences of float in the actual code (ignoring legit uses in comments and printouts).

What complicates things furthermore, is that there are some (albeit few) legit uses of float in the code payload (so we are really looking for a way to identify all places that require manual inspection, rather than run some automatic search-and-replace.)

the code also contains C-style static casts to (float), so relying on compiler warnings to identify type mismatches is often not an option.

the code base consists of more than 3000 (C and C++) files accumulating about 750000 lines of code.

the code is cross-platform (linux, osx, w32 being the main targets; but also freebsd and similar), and is compiled with the various native compilers (gcc/g++, clang/clang++, VisualStudio,...).

so far...

so far I'm using something ugly like:

 grep "\bfloat\b" | sed -e 's|//.*||' -e 's|"[^"]*"||g' | grep "\bfloat\b"

but I'm thinking that there must be some better way to search only payload code.

Answer 1

IMHO there is a good answers on a similar question at "Unix & Linux":

grep works on pure text and does not know anything about the underlying syntax of your C program. Therefore, in order not search inside comments you have several options:

1. Strip C-comments before the search, you can do this using gcc -fpreprocessed -dD -E yourfile.c For details, please see Remove comments from C/C++ code

2. Write/use some hacky half-working scripts like you have already found (e.g. they work by skipping lines starting with // or /*) in order to handle the details of all possible C/C++ comments (again, see the previous link for some scary testcases). Then you still may have false positives, but you do not have to preprocess anything.

3. Use more advanced tools for doing "semantic search" in the code. I have found "coccigrep": http://home.regit.org/software/coccigrep/ This kind of tools allows search for some specific language statements (i.e. an update of a structure with given name) and certainly they drop the comments.

https://unix.stackexchange.com/a/33136/158220

Although it doesn't completely cover your "not in strings" requirement.

Answer 2

It might practically depend upon the size of your code base, and perhaps also on the editor you are usually using. I am suggesting to use GNU emacs (if possible on Linux with a recent GCC compiler...)

For a small to medium size code (e.g. less than 300KLOC), I would suggest using the grep mode of Emacs. Then (assuming you have bound the next-error Emacs function to some key, perhaps with (global-set-key [f10] 'next-error) in your ~/.emacs...) you can quickly scan every occurrence of float (even inside strings or comments, but you'll skip very quickly such occurrences...). In a few hours you'll be done with a medium sized source code (and that is quicker than learning how to use a new tool).

For a large sized code (millions of lines), it might be worthwhile to customize some static analysis tool or compiler. You could use GCC MELT to customize your GCC compiler on Linux. Its findgimple mode could be inspirational, and perhaps even useful (you probably want to find all Gimple assignments targeting a float)

BTW, you probably don't want to replace all occurrences -but only most of them- of the float type with double (probably suitably typedef-ed...), because very probably you are using some external (or standard) functions requiring a float.

The CADNA tool might also be useful, to help you estimate the precision of results (so help you deciding when using double is sensible).

Using semantical tools like GCC MELT, CADNA, Coccinelle, Frama-C (or perhaps Fluctuat, or Coccigrep mentioned in g0hl1n's answer) would give more precise or relevant results, at the expense of having to spend more time (perhaps days!) in learning and customizing the tool.

Answer 3

The robust way to do this should be with cscope (http://cscope.sourceforge.net/) in line-oriented mode using the find this C symbol option but I haven't used that on a variety of C standards so if that doesn't work for you or if you can't get cscope then do this:

find . -type f -print |
while IFS= read -r file
do
    sed 's/a/aA/g; s/__/aB/g; s/#/aC/g' "$file" |
    gcc -P -E - |
    sed 's/aC/#/g; s/aB/__/g; s/aA/a/g' |
    awk -v file="$file" -v OFS=': ' '/\<float\>/{print file, $0}'
done

The first sed replaces all hash (#) and __ symbols with unique identifier strings, so that the preprocessor doesn't do any expansion of #include, etc. but we can restore them after preprocessing.

The gcc preprocesses the input to strip out comments.

The second sed replaces the hash-identifier string that we previously added with an actual hash sign.

The awk actually searches for float within word-boundaries and if found prints the file name plus the line it was found on. This uses GNU awk for word-boundaries \< and \>.

The 2nd sed's job COULD be done as part of the awk command but I like the symmetry of the 2 seds.

Unlike if you use cscope, this sed/gcc/sed/awk approach will NOT avoid finding false matches within strings but hopefully there's very few of those and you can weed them out while post-processing manually anyway.

It will not work for file names that contain newlines - if you have those you can put the body in a script and execute it as find .. -print0 | xargs -0 script.

Modify the gcc command line by adding whatever C or C++ version you are using, e.g. -ansi.