multiple targets from one recipe and parallel execution

Question

I have a project which includes a code generator which generates several .c and .h files from one input file with just one invocation of the code generator. I have a rule which has the .c and .h files as multiple targets, the input file as the prerequisite, and the recipe is the invocation of the code generator. I then have further rules to compile and link the generated .c files.

This works fine with a -j factor of 1, but if I increase the j factor, I find I get multiple invocations of the code generator, up to the -j factor or the number of expected target files, whichever is smallest. This is bad because multiple invocations of the code generator can cause failures due to the generated code being written multiple times.

I'm not going to post my actual (large) code here, but I have been able to construct a small example which appears to demonstrate the same behavior.

The Makefile looks like this:

output.concat: output5 output4 output3 output2 output1
    cat $^ > $@

output1 output2 output3 output4 output5: input
    ./frob input

clean:
    rm -rf output*

Instead of a code generator, for this example I have written a simple shell script, frob which generates multiple output files from one input file:

#!/bin/bash

for i in {1..5}; do
    {
    echo "This is output${i}, generated from ${1}. input was:"
    cat ${1}
    } > output${i}
done

When I run this Makefile with non-unity -j factors, I get the following output:

$ make -j2 
./frob input
./frob input
cat output5 output4 output3 output2 output1 > output.concat
$

We see ./frob here gets invoked twice, which is bad. Is there some way I can construct this rule such that the recipe only gets invoked once, even with a non-unity -j factor?

I have considered changing the rule so that just one of the expected output files is the target, then adding another rule with no recipe such that its targets are the remaining expected output files, and the prerequisite is the first expected output file. But I'm not sure this would work, because I don't know if I can guarantee the order in which the files are generated, and thus may end up with circular dependencies.

Answer 1

This is how make is defined to work. A rule like this:

foo bar baz : boz ; $(BUILDIT)

is exactly equivalent, to make, to writing these three rules:

foo : boz ; $(BUILDIT)
bar : boz ; $(BUILDIT)
baz : boz ; $(BUILDIT)

There is no way (in GNU make) to define an explicit rule with the characteristics you want; that is that one invocation of the recipe will build all three targets.

However, if your output files and your input file share a common base, you CAN write a pattern rule like this:

%.foo %.bar %.baz : %.boz ; $(BUILDIT)

Strangely, for implicit rules with multiple targets GNU make assumes that a single invocation of the recipe WILL build all the targets, and it will behave exactly as you want.

Answer 2

Correctly generate and update multiple targets a b с in parallel make -j from input files i1 i2:

all: a b c
.INTERMEDIATE: d
a: d
b: d
c: d
d: i1 i2
    cat i1 i2 > a 
    cat i1 i2 > b
    cat i1 i2 > c

• If any of a,b,c are missing, the pseudo-target d is remade. The file d is never created; the single rule for d avoids several parallel invocations of the recipe.

• .INTERMEDIATE ensures that missing file d doesn't trigger the d recipe.

• Some other ways for multiple targets in the book "John Graham-Cumming - GNU Make Book" p.92-96.

Answer 3

@MadScientist's answer is promising - I think I could possibly use that. In the meantime, I have been playing with this some more and come up with a different possible solution, as hinted at in the question. I can split the rule in two as follows:

INPUT_FILE = input
OUTPUT_FILES = output5 output4 output3 output2 output1
OUTPUT_FILE1 = $(firstword $(OUTPUT_FILES))
OUTPUT_FILES_REST = $(wordlist 2,$(words $(OUTPUT_FILES)),$(OUTPUT_FILES))

$(OUTPUT_FILE1): $(INPUT_FILE)
    ./frob $<
    touch $(OUTPUT_FILES_REST)

$(OUTPUT_FILES_REST): $(OUTPUT_FILE1)

Giving only one output file as a target fixes the possible parallelism problem. Then we make this one output file as a prerequisite to the rest of the output files. Importantly in the frob recipe, we touch all the output files with the exception of the first so we are guaranteed that the first will have an older timestamp than all the rest.

Answer 4

As of make 4.3 (Jan 2020) make allows grouped targets. As per docs the following will update all targets only once if any of the targets is missing or outdated:

foo bar biz &: baz boz
        echo $^ > foo
        echo $^ > bar
        echo $^ > biz

Answer 5

Answer by Ivan Zaentsev almost worked for me, with exception of the following issue. Only when running parallel make (-j2 or above), when a prerequisite of the generated file was changed, the generated file was regenerated successfully, however, the subsequent targets that depend on the generated file were not rebuilt.

The workaround I found was to provide a recipe for the generated files (the trivial copy command), besides the dependency on the intermediate target (d):

d: i1 i2
    cat i1 i2 > a.gen 
    cat i1 i2 > b.gen
    cat i1 i2 > c.gen
.INTERMEDIATE: d
a.gen : d
b.gen : d
c.gen : d

a: a.gen d
    cp $< $@
b: b.gen d
    cp $< $@
c: c.gen d
    cp $< $@

e: a b c
    some_command $@ $^

The clue was this debug output from make when running without the workaround (where 'e' was not rebuilt with make -j2, despite a,b,c being rebuilt):

       Finished prerequisites of target file `a'.
       Prerequisite `d' of target `a' does not exist.
      No recipe for `a' and no prerequisites actually changed.
      No need to remake target `a'.

Answer 6

Here is the solution that seemed to work for me (credit to @Ivan Zaentsev for the main solution and to @alexei for pointing out the problem with it). It is similar to the original approach with one major change. Instead of generating temporary files (*.gen as suggested), it just touches the files that depend on the INTERMEDIATE file. :

default: f

.INTERMEDIATE: temp
a b c: temp
    touch $@

temp: i1 i2
    echo "BUILD: a b c"
    cat i1 i2 > a
    cat i1 i2 > b
    cat i1 i2 > c

e: a b c
    echo "BUILD: e"
    touch $@

f: e
    echo "BUILD: f"
    touch $@