5

I am using a third party open source project and need to strip out the inactive #ifs, #ifdefs, etc to better understand the code flow.

Is there a way to use make to produce versions of the source files without these directives? I'd like to avoid expanding macros, just remove directives.

I was looking at https://gcc.gnu.org/onlinedocs/gcc/Preprocessor-Options.html

and it seems like -dD and -fdirectives-only are good options to start.

Where will these preprocessed files appear? Where do I add these commands for use with a Makefile and "make"? I tried running "make -n" to produce a script and adding options to the g++ and gcc calls in the script after -Wformat among other things, but I dont notice anything.

I'm not sure if this complicates anything, but I am also using avr-gcc and avr-g++.

I have looked at coan, which does not support #included #defines so it would not work for this purpose, and I could not get sunifdef to work. Is there is a way of doing this with the preprocessor.

The defines are scattered among the current file, the included files, and included makefiles that specify -Dfoo=opt options.

user221237
  • 143
  • 1
  • 8
  • 3
    You can fork a library and get the "only works on my machine" version of an open source project. But don't expect anybody to support or contribute to it after you are done, they sweated bullets putting those #ifdefs into the code. A bit like teenage sex, one mistake and you'll have to support it for the rest of your life :) – Hans Passant Aug 01 '14 at 22:19
  • Yes, but it can be pretty difficult to understand what is going on when the flow is obscured by countless directives. – user221237 Aug 01 '14 at 22:22
  • Don't know what `sunifdef` is, probably some fork of `unifdef`. Why exactly doesn't it help you? – n. m. could be an AI Aug 01 '14 at 22:27
  • 3
    What is it about sunifdef and unifdef that makes them not work for you? This is exactly what they are for. – skrrgwasme Aug 01 '14 at 22:42
  • "I could not get sunifdef to work" -- So you already have an answer to your question. The correct question should be aimed at getting it to work. – Jim Balter Aug 01 '14 at 22:59
  • The question is whether this can be done with the gcc preprocessor, not whether other utilities might be able to do this. – user221237 Aug 01 '14 at 23:00
  • Your question is not "whether this can be done", it's "How do I ...", and the answer is *mu*, since that's not a capability of the preprocessor. Use the right tool. – Jim Balter Aug 01 '14 at 23:01
  • At the least, consult [Is there a C pre-processor which eliminates #ifdef blocks based on values defined/undefined?](http://stackoverflow.com/questions/525283/is-there-a-c-pre-processor-which-eliminates-ifdef-blocks-based-on-values-define) I'm tempted to close this as a duplicate of that. Note, in particular, that `sunifdef` and `coan` do not include the contents of all the headers in the output. – Jonathan Leffler Aug 29 '14 at 18:35

3 Answers3

5

You're on the right track with your preprocessor options. -D will define a macro with a value of 1, -U will cancel any previous definition (it will become undefined), and -fdirectives-only will suppress macro expansion. In addition to those, you can use the -E flag with gcc to tell it to provide the preprocessor output as separate files for your examination. However, I don't think they're going to be quite what you expect. The CPP (C pre-processor) output may have other things added to it, as suggested by this SO question, and you should check the gnu CPP output manual page. That is what you will get from the CPP.

It sounds like you want to be able to strip this extraneous code once and develop from there. To do that, I would encourage you to give unifdef another try. This is what unifdef was designed to do, while the CPP was designed to prepare code for compilation. They're different tasks, so you should use the right tools for them. It is available as a standalone application at http://dotat.at/prog/unifdef/ and is built into some Linux Shells.

It allows you to specify macros that you want it to consider defined or undefined, and it removes blocks of code where the conditional directive would evaluate to false. For example, you can run it like this:

unifdef -I< path > -DMACRO1 -UMACRO2

It will search through the directory specified by < path > through C/C++ source files, looking for #if, #ifdef, #ifndef, etc. When it encounters them, it will evaluate the conditional expression and selectively remove the code controlled by that expression. Consider an input file with this code:

int i = 0;
#ifdef MACRO1
int j = 0;
#endif /* ifdef MACRO1 */
int k = 0;

int m = 0;
#if (MACRO1 && MACRO2)
int n = 0;
#endif /* if (MACRO1 && MACRO2) */
int p = 0;

int q = 0;
#ifdef MACRO3
int r = 0;
#endif /* ifdef MACRO3 */
int t = 0;

If we call unifdef like my example above, the output will be this:

int i = 0;
int j = 0;
int k = 0;

int m = 0;
int p = 0;

int q = 0;
#ifdef MACRO3
int r = 0;
#endif /* ifdef MACRO3 */
int t = 0;

Notice that the declaration of n has been removed, because it was contained in a preprocessor #if/#endif block whose controlling expression evaluated to false (we told unifdef to consider MACRO2 undefined). The declaration of j remains, but the #ifdef and #endif statements were removed because the controlling expression was known to be true.

The block that depends on MACRO3 is left untouched because its state is unknown.

There is a significant amount of flexibility and control over how this runs, too.

If you decided you do want it to be part of your build process, you can always add it to your makefile.

If you do not have a list of which macros should be defined or undefined available, you can use the "unifdefall" script provided with unifdef and it will use the CPP to discover macro definitions in the source code on its own, and remove/keep code blocks according to the definitions contained in the source code.

TL;DR

Yes you can (sort of) do it with the preprocessor. But unifdef and sunifdef are tools that are made to do exactly this, so you should use them instead.

Community
  • 1
  • 1
skrrgwasme
  • 9,358
  • 11
  • 54
  • 84
  • Thanks for taking the time to type this up Scott! I worry unifdef might produce something that is not consistent with the CPP. Will unifdef handle dependency graphs (if something gets #def'd, then #undef'd, then the order of dependency graph traversal is relevant). Will unifdef handle implicit defines that come with the machine you're compiling on? Also, the Makefile includes many other makefiles, each of which is specifying -D options, so I'd have to go through each of these dependent makefiles and create the -D list by hand which is somewhat risky. – user221237 Aug 01 '14 at 23:16
  • I'm not sure about the define then undefine (or vice-versa) order. You'll have to check the documentation. It's not something I've had to worry about before. As for the makefile -D options, there is some risk to it, but you could grep through your makefiles for -D, pipe the output to a text file, and use that file as unifdef input. It can read definitions from text files and uses the same -D / -U argument formats as GCC does, and you only have to do it once if you save the file. An automated method like that should minimize the danger of screwing it up. – skrrgwasme Aug 01 '14 at 23:24
  • It would be nice if it were that easy, but the makefiles use makefile $VARIABLES, which are assigned based on conditions in the make file. Then specific variables are concatenated with -Ds in front in a separate step. Is there a way to get a list of all the defines using the preprocessor, then run unifdef with that as input, or at this point is it easier to try to clean up the preprocessor output to get the original files back without the defines? – user221237 Aug 01 '14 at 23:34
  • You can use flags [like this](http://stackoverflow.com/q/2224334/2615940) to get the #define list out of the preprocessor, but you're correct that this leads down the path of diminishing returns, where you have to wonder if just cleaning up the preprocessor output files would be easier. Honestly, I don't know because I haven't had to deal with the makefile mess you currently have. I'd suggest dumping the preprocessor output files and make a rough estimate of the time sink cleaning them would be and compare that to the effort you know it will take to for the unifdef route. – skrrgwasme Aug 01 '14 at 23:37
  • Unifdef is absolutely the tool to use. You can validate the job it does by producing pre-processed output before and after running unifdef and comparing them. They should be identical, give or take a bit of white space. – david.pfx Aug 01 '14 at 23:40
  • @david.pfx It is, but through this comment thread, I now see the complexities the OP is facing with obtaining a list of the macros that would be input to unifdef because they're buried in a mess of makefiles that makes them difficult to extract. – skrrgwasme Aug 01 '14 at 23:42
  • @user221237 Here's one more detail for you: you can use the -dM flag with GCC to get a list of the macro definitions it is using. Try to construct an unifdef input file from that. Also, read the bottom of the unifdef man page - it has a blurb about an "unifdefall" script that may interest you. – skrrgwasme Aug 02 '14 at 00:04
  • Yeah, going with preprocessor or preproc/unifdef are both good solutions. Do you know how to add the preprocessor options when using make? Since the Makefile is just a bunch of includes of many other makefiles and it was not clear where to add these make options in the makefile, so I used make -n to generate the actual shell commands make would run and tried adding -dD to the gcc and g++ commands, and also tried doing make CPPFLAGS="-dD" among other things suggested on other SO posts. Do you know where I should be looking for the preprocessor output. – user221237 Aug 02 '14 at 01:19
  • @ScottLawson The unifdefall is useful. Do you know how to do this with a makefile instead of invoking cpp directly, which does not use a makefile? I think it might be possible to do with make -n to get the underlying gcc command and replacing the gcc command with something, but I am not sure how to do it since I am not super skilled with makefiles. – user221237 Aug 02 '14 at 01:55
  • 1
    @user221237 This is getting beyond my knowledge as well now. I'm not an experienced makefile user and haven't had to invoke preprocessor-only options before, so I'm not sure how to go about doing it. If you can't get it to work on your own, I would suggest opening up a new SO question inquiring *specifically* about getting the -dM flag to work through a makefile. Once you have that set though, I think getting unifdef to work for you will be a trivial matter. – skrrgwasme Aug 02 '14 at 02:45
2

Assumptions

  1. The aim of the exercise is to produce a body of C/C++ source code with most of the conditional compilation removed, and which compiles to the identical binaries.
  2. This is third party source code, and you are aware of the problems of merging subsequent updates.
  3. This is open source, but you have no intention of ever distributing modified source code.
  4. The programs are arbitrarily complex and are built by arbitrarily complex makefiles or similar tools, with command-line symbol definitions and/or configuration include files.

My strategy is to use a program like unifdef. The first time I did this I wrote my own, and you may have to modify the program to produce the desired results.

The core strategy is:

  1. Identify a single likely defined symbol (experimentation or trial and error required).
  2. Run the code through unifdef.
  3. Optionally, compare before and after source visually to spot obvious problems.
  4. Build the after version to ensure it builds correctly.
  5. Compile the before and after versions to produce pre-processed output using the same makefiles.
  6. Compare pairs of before and after pre-processed source. They should be identical, give or take some white space.
  7. Resolve issues by editing either before or after version as required.
  8. Optionally, remove all references to the symbol from all makefiles. [It should make no difference.]
  9. Repeat, using the after version and a different symbol.

One symbol at a time, testing thoroughly every time. Some symbols may turn out to be too hard, and if you have much more than a million lines of source code and a hundred or so symbols it can all get out of hand.

Final step: if you modify unifdef then feel free to contribute your changes back to the community. This is a seriously challenging task to do well!

david.pfx
  • 10,520
  • 3
  • 30
  • 63
  • I wrote a script that goes 95% of the way and you fix the last 5% manually. The first step is to use make -n > mk.sh, add -E -dM to the underlying g++ call in mk.sh to get a list of defines used to link the object. Cat these defines to the top of every file, run unifdef on the file, then remove those cat'd defines. It will not work for include guards or includes like that (#ifndef a #define a), so i scripted something with seds/greps to fix this corner case. It was a 6 hour ordeal, but the 40,000 line change compiles/runs – user221237 Aug 05 '14 at 19:56
  • If you get out of that for something like a day's work you've done OK. Hopefully this will help someone else with he same problem. – david.pfx Aug 05 '14 at 22:58
1

Use make -n to create the shell script produced by the makefile. Go to the line where it runs avr-g++ and add -dM -E before all the rest of the options. Go to the file after the -o and the list of #defines will be there (it should probably be something.o) Use unifdef -f definesFile.o filename

user221237
  • 143
  • 1
  • 8