GNU make: Generating automatic dependencies with generated header files

Question

So I followed the Advanced Auto-Dependency Generation paper --

Makefile:

SRCS := main.c foo.c

main: main.o foo.o

%.o: %.c
    $(CC) -MMD -MG -MT '$@ $*.d' -c $< -o $@
    cp $*.d $*.tmp
    sed -e 's;#.*;;' -e 's;^[^:]*: *;;' -e 's; *\\$$;;' \
        -e '/^$$/d' -e 's;$$; :;' < $*.tmp >> $*.d
    rm $*.tmp

clean::
    -rm *.o *.d main

-include $(SRCS:.c=.d)

main.c:

#include "foo.h"

int main(int argc, char** argv) {
  foo() ;
  return 0 ;
}

foo.h:

#ifndef __FOO_H__
#define __FOO_H__

void foo() ;

#endif

-- and it works like a charm.

---

But when foo.h becomes a generated file --

Makefile:

...

HDRS := foo.h

$(HDRS):
    mk_header.sh $*

clean::
    -rm $(HDRS)
...

mk_header.sh:

#!/bin/bash
UP=$(tr "[:lower:]" "[:upper:]" <<< $1)

cat <<EOF > $1.h
#ifndef __${UP}_H__
#define __${UP}_H__

void $1() ;

#endif
EOF

The 1st time I run make, main.d is not yet generated, and thus foo.h is not considered a prerequisite, and thus isn't been generated:

$ ls
foo.c  main.c  Makefile  mk_header.sh*

$ make
cc -MMD -MG -MT 'main.o main.d' -c main.c -o main.o
cp main.d main.tmp
sed -e 's;#.*;;' -e 's;^[^:]*: *;;' -e 's; *\\$;;' \
    -e '/^$/d' -e 's;$; :;' < main.tmp >> main.d
rm main.tmp
cc -MMD -MG -MT 'foo.o foo.d' -c foo.c -o foo.o
cp foo.d foo.tmp
sed -e 's;#.*;;' -e 's;^[^:]*: *;;' -e 's; *\\$;;' \
    -e '/^$/d' -e 's;$; :;' < foo.tmp >> foo.d
rm foo.tmp
cc   main.o foo.o   -o main

$ ls
foo.c  foo.d  foo.o  
main*  main.c  main.d  main.o  
Makefile  mk_header.sh*

Only in the 2nd invocation of make, the foo.h is generated, and as a result another build cascades.

$ make
./mk_header.sh foo
cc -MMD -MG -MT 'main.o main.d' -c main.c -o main.o
cp main.d main.tmp
sed -e 's;#.*;;' -e 's;^[^:]*: *;;' -e 's; *\\$;;' \
    -e '/^$/d' -e 's;$; :;' < main.tmp >> main.d
rm main.tmp
cc   main.o foo.o   -o main

$ ls
foo.c  foo.d  foo.h  foo.o  
main*  main.c  main.d  main.o  
Makefile  mk_header.sh*

And only after that make realizes that:

$ make
make: `main' is up to date.

---

So my question is: Is there a way to extend the recipe suggested by the paper above, to allow for generated header files, without the elimination of the performance gain realized by not having to re-evaluate the entire make tree when including the *.d fragments?

Answer 1

The problem is that the *.d Makefile-fragments generation must be performed after all the header generation is complete. Putting it this way, one can use the make dependencies to force the right order:

SRCS := main.c foo.c
HDRS := foo.h

main: main.o foo.o

%.o: %.c | generated_headers
    $(CC) -MMD -MG -MT '$@ $*.d' -c $< -o $@
    cp $*.d $*.tmp
    sed -e 's;#.*;;' -e 's;^[^:]*: *;;' -e 's; *\\$$;;' \
        -e '/^$$/d' -e 's;$$; :;' < $*.tmp >> $*.d
    rm $*.tmp

-include $(SRCS:.c=.d)

$(HDRS):
    mk_header.sh $*

generated_headers: $(HDRS)

clean:
    -rm $(HDRS) *.o *.d main

.PHONY: clean generated_headers

Notes:

1. I use an order-only dependency.

2. This solution is fairly scalable: Each generate-header rule, needs only to be a prerequisite of the generated_headers .PHONY target. Assuming that the header generation rule is written properly, once it has been generated correctly, satisfying the generated_headers target should be a no-op.

3. One can't compile a single object, even if that object does not require any generated headers, without generating all the generated headers of the project first. While this is technically sound, your developers will complain.

   So you should think about having a FAST_AND_LOOSE flag, that will turn this feature off:

   %.o: %.c | $(if $(FAST_AND_LOOSE),,generated_headers)
       ...
   

   Thus a developer may issue:

   make FAST_AND_LOOSE=1 main.o
   

Answer 2

The makefile in the original question doesn't work for me with gcc 4.8.2:

cc -MMD -MG -MT main.d -c main.c -o main.o
cc1: error: -MG may only be used with -M or -MM

I guess gcc changed the behaviour of -MG at some point in the last 4 years.

It seems that if you want to support generated header files, there is no longer any way to generate the ".d" file and the ".o" file at the same time, without invoking the C preprocessor twice.

So I've updated the recipe to:

%.o: %.c
    $(CC) -MM -MG -MP -MT $*.o -MF $*.d $<
    $(CC) -c $< -o $@

(Note also that gcc now has -MP to generate phony targets for each header, so you no longer need to run sed on gcc's output.)

We still have the same problem as the original question -- running make the first time fails to generate foo.h:

$ make
cc -MM -MG -MP -MT main.o -MF main.d main.c
cc -c main.c -o main.o
main.c:1:17: fatal error: foo.h: No such file or directory
 #include "foo.h"
                 ^
compilation terminated.
Makefile:7: recipe for target 'main.o' failed
make: *** [main.o] Error 1

Running it again works:

$ make
./mk_header.sh foo
cc -MM -MG -MP -MT main.o -MF main.d main.c
cc -c main.c -o main.o
cc   main.o   -o main

Since we have to run the C preprocessor twice anyway, let's generate the .d file in a separate rule:

%.d: %.c
    $(CC) -MM -MG -MP -MT $*.o -MF $@ $<

%.o: %.c
    $(CC) -c $< -o $@

Now it generates the header file correctly:

$ make clean
rm -f *.o *.d main foo.h
$ make
cc -MM -MG -MP -MT main.o -MF main.d main.c
./mk_header.sh foo
cc -c main.c -o main.o
cc   main.o   -o main

Does this suffer from the performance issue that the original question was trying to avoid? This is essentially the "Basic Auto-Dependencies" solution described in the Advanced Auto-Dependency Generation paper.

That paper claims 3 problems with this solution:

1. We re-exec make if anything changes.
2. Ugly but harmless warning: "main.d: No such file or directory"
3. Fatal error "no rule to make targe foo.h" if foo.h file is removed, even if mention of it is removed from the .c file.

Problem 2 is solved by using -include instead of include. As far as I can tell, this is orthogonal to the paper's technique for avoiding re-exec of make. At least I haven't been able to cause any problems by using -include instead of include.

Problem 3 is solved by GCC's -MP (or the equivalent sed script) -- this is also orthogonal to the technique for avoiding re-exec of make.

Problem 1 can be perhaps ameliorated somewhat by something like this:

%.d: %.c
    $(CC) -MM -MG -MP -MT $*.o -MF $@.new $<
    cmp $@.new $@ 2>/dev/null || mv $@.new $@; rm -f $@.new

Before that change:

$ make clean
rm -f *.o *.d main foo.h
$ make -d 2>&1 | grep Re-executing
Re-executing[1]: make -d
$ make -d 2>&1 | grep Re-executing
$ touch main.c; make -d 2>&1 | grep Re-executing
Re-executing[1]: make -d

After that change:

$ make clean
rm -f *.o *.d main foo.h
$ make -d 2>&1 | grep Re-executing
Re-executing[1]: make -d
$ make -d 2>&1 | grep Re-executing
$ touch main.c; make -d 2>&1 | grep Re-executing

Slightly better. Of course if a new dependency is introduced, make will still need to re-execute. Maybe there's nothing that can be done to improve this; it seems to be a tradeoff between correctness and speed.

All of the above was tested with make 3.81.

Answer 3

Short answer: no. The recipe described in the paper is very clever, one of my favorites, but it's a sophisticated use of a crude tool. It takes advantage of the usual scheme in which all needed headers exist; what it tries to solve is the problem of determining which headers, if recently modified, require the given object file to be rebuilt. In particular, if the object file doesn't exist then it must be rebuilt-- and in that case there's no reason to worry about the header files because the compiler will surely find them.

Now header files are generated. So foo.h may not exist, so somebody will have to run the script to generate it, and only Make can do that. But Make can't know that foo.h is necessary without performing some analysis of main.c. But that really can't happen until Make starts to execute main-related rules (e.g main.o or main.o.d), which it cannot execute until after it has decided which targets it is going to build.

So we will have to use... recursive make! [Dun-dun-dunnnn!]

We can't achieve the paper's goal of avoiding reinvocation of Make, but we can at least avoid (some) unnecessary rebuilding. You could do something like the "Basic Auto-Dependencies" described in the paper; the paper describes the problems of that approach. Or you could use a command like the one in the "Advanced" recipe to generate a list of headers, then pass that to $(MAKE); this approach is tidy, but might call Make many times on the same header, depending on what your code tree looks like.

Answer 4

You could create an explicit dependency rule for your generated header:

main.o: foo.h

If the generated header is directly included in a small number of files, this may be a workable approach.