5

I have a tool which generates multiple output files, which is notoriously hard to model in make. I'm using the recipe at GNU Makefile rule generating a few targets from a single source file, which seems simple and reliable. And it almost works.

Unfortunately, I'm seeing some very odd behaviour with regard to parallel builds, where it seems to be dropping dependencies sometimes; and I don't understand why.

Here's my test case:

out3: out1 out2
    touch out3

.INTERMEDIATE: out.intermediate
out1 out2: out.intermediate
out.intermediate: in
    touch out1 out2

If I build it once, it works:

$ touch in
$ make -f test.mk out3 -j4
touch out1 out2
touch out3

out1 and out2 are built together, once, which is good; then out3 is built from the result.

Now I touch the input file, simulating an incremental build, and try again:

$ touch in
$ make -f test.mk out3 -j4
touch out1 out2

That's rebuild out1 and out2, correctly... but it hasn't rebuilt out3, which it should have. Then if I do another build:

$ make -f test.mk out3 -j4
touch out3

...and it catches up.

This only applies to parallel builds. -j1 builds work fine.

This is bad --- I need to be able to rely on correct builds. Does anyone have any idea as to what's going on?

This is GNU Make 4.1.

Community
  • 1
  • 1
David Given
  • 13,277
  • 9
  • 76
  • 123
  • The recipe you are referring to is the non-GNU Make fallback proposal in the 2nd most upvoted answer to the question. As you *are* using GNU Make, you could use the GNU Make solution in the same answer, or the GNU Make solution in the most upvoted answer: I tried that one on your test case and it's good. – Mike Kinghan Jun 17 '16 at 16:09
  • Which GNU make solution? Do you mean patterns? That's not appropriate here as my filenames aren't related. – David Given Jun 17 '16 at 23:30
  • The necessary relatedness is that the target filenames (not the prerequisite filename) have at least one character in common, even '.' Not even that? – Mike Kinghan Jun 18 '16 at 08:01
  • I cannot guarantee that, no. However, that's a side issue; I'm mainly concerned about what looks suspiciously like an outright bug in the dependency engine. Is this *supposed* to work, or does `.INTERMEDIATE` not behave the way I think? – David Given Jun 19 '16 at 05:29

1 Answers1

14

The SO answer you link to has an error; surprising that so many people have apparently used it with success.

What's happening (you can add the -rRd options to your make invocation to see in more detail) is a result of GNU make's directory caching. GNU make doesn't expect the filesystem state to change in any way other than what your makefile describes, and so it caches the contents of directories to provide a significant performance increase (for large makefiles/directories).

Basically, when make runs the rule:

out.intermediate: in
        touch out1 out2

it doesn't expect that this recipe will update any targets other than the one listed in the rule: out.intermediate. If the other files out1 and out2 are already internalized in the directory cache (which they will be since they exist and we've already checked them as prerequisites to out3), then make won't go back to the filesystem and see if they've been updated or not: make "knows" they can't have changed because no rule that it ran could have changed them, according to the makefile.

There's a simple, one-character fix to make all this work. Change this line:

out1 out2: out.intermediate

to this:

out1 out2: out.intermediate ;

If you want to be more explicit you could also use this:

out1 out2: out.intermediate
        @:

or even this, for debugging:

out1 out2: out.intermediate
        @echo Do nothing

What all these have in common is that you've now defined not just a dependency relationship between the targets and prerequisite, but you've also given a recipe that make should invoke for that rule. Even if the recipe is empty (as in the first example), so make doesn't actually run any commands, make will still infer that it's possible that the timestamps on out1 and/or out2 have changed, and it will invalidate the cached modified time for those targets and re-retrieve them from the filesystem.

MadScientist
  • 92,819
  • 9
  • 109
  • 136