123

I am attempting to do the following. There is a program, call it foo-bin, that takes in a single input file and generates two output files. A dumb Makefile rule for this would be:

file-a.out file-b.out: input.in
    foo-bin input.in file-a.out file-b.out

However, this does not tell make in any way that both targets will be generated simultaneously. That is fine when running make in serial, but will likely cause trouble if one tries make -j16 or something equally crazy.

The question is whether there exists a way to write a proper Makefile rule for such a case? Clearly, it would generate a DAG, but somehow the GNU make manual does not specify how this case could be handled.

Running the same code twice and generating only one result is out of the question, because the computation takes time (think: hours). Outputting only one file would also be rather difficult, because frequently it is used as an input to GNUPLOT which doesn't know how to handle only a fraction of a data file.

makesaurus
  • 1,393
  • 3
  • 10
  • 7
  • 2
    Automake has documentation on this: https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html – Frank Aug 09 '18 at 15:33
  • As of GNU make version 4.3, this is supported by grouped targets. You only need to change `:` for `&:`. See my answer below for more details. – jonas-schulze Dec 07 '22 at 16:00

9 Answers9

137

The trick is to use a pattern rule with multiple targets. In that case make will assume that both targets are created by a single invocation of the command.

all: file-a.out file-b.out
file-a%out file-b%out: input.in
    foo-bin input.in file-a$*out file-b$*out

This difference in interpretation between pattern rules and normal rules doesn't exactly make sense, but it's useful for cases like this, and it is documented in the manual.

This trick can be used for any number of output files as long as their names have some common substring for the % to match. (In this case the common substring is ".")

slowdog
  • 6,076
  • 2
  • 27
  • 30
  • 4
    This actually isn't correct. Your rule specifies that files matching these patterns are to be made from input.in using the command specified, but nowhere does it say that they are made simultaneously. If you actually run it in parallel, `make` will run the same command twice simultaneously. – makesaurus Jun 25 '10 at 18:32
  • 53
    Please try it before you say it doesn't work ;-) The GNU make manual says: "Pattern rules may have more than one target. Unlike normal rules, this does not act as many different rules with the same prerequisites and commands. If a pattern rule has multiple targets, make knows that the rule's commands are responsible for making all of the targets. The commands are executed only once to make all the targets." http://www.gnu.org/software/make/manual/make.html#Pattern-Intro – slowdog Jun 26 '10 at 14:47
  • 4
    But what if I have completely different inputs?, Say foo.in and bar.src? Does pattern rules support empty pattern match? – Grwlf Jul 25 '13 at 13:32
  • 1
    grwlf: The same part of the manual I linked to above says "the ‘%’ matches any *nonempty* substring". So completely different input names won't work. – slowdog May 15 '14 at 11:19
  • Any way to do this when the files don't share a prefix? – dcow Jun 19 '15 at 04:53
  • 3
    @dcow: The output files only need to share a substring (even a single character, such as ".", is enough) not necessarily a prefix. If there's no common substring, you can use an intermediate target as described by richard and deemer. @grwlf: Hey, that means "foo.in" and "bar.src" can be done: `foo%in bar%src: input.in` :-) – slowdog Jun 22 '15 at 10:54
  • @slowdog indeed in my situation the do not share anything but a `.`. Clever. – dcow Jun 22 '15 at 18:40
  • 2
    If the respective output files are kept in variables the recipe can be written as: `$(subst .,%,$(varA) $(varB)): input.in` (tested with GNU make 4.1). – stefanct Mar 23 '17 at 10:21
  • The benefit is that you don't need a .PHONY target to bundle the execution, since that .PHONY target is always executed. Very nice! – bebbo Jan 05 '18 at 22:58
  • Works well with sequential make, but will still start `N` processes for the same target with `make -jN`. – cmaster - reinstate monica Jul 26 '18 at 11:35
63

Make doesn't have any intuitive way to do this, but there are two decent workarounds.

First, if the targets involved have a common stem, you can use a prefix rule (with GNU make). That is, if you wanted to fix the following rule:

object.out1 object.out2: object.input
    foo-bin object.input object.out1 object.out2

You could write it this way:

%.out1 %.out2: %.input
    foo-bin $*.input $*.out1 $*.out2

(Using the pattern-rule variable $*, which stands for the matched part of the pattern)

If you want to be portable to non-GNU Make implementations or if your files can't be named to match a pattern rule, there is another way:

file-a.out file-b.out: input.in.intermediate ;

.INTERMEDIATE: input.in.intermediate
input.in.intermediate: input.in
    foo-bin input.in file-a.out file-b.out

This tells make that input.in.intermediate won't exist before make is run, so its absence (or its timestamp) won't cause foo-bin to be run spuriously. And whether either file-a.out or file-b.out or both are out-of-date (relative to input.in), foo-bin will be only run once. You can use .SECONDARY instead of .INTERMEDIATE, which will instruct make NOT to delete a hypothetical file name input.in.intermediate. This method is also safe for parallel make builds.

The semicolon on the first line is important. It creates an empty recipe for that rule, so that Make knows we will really be updating file-a.out and file-b.out (thanks @siulkiulki and others who pointed this out)

deemer
  • 1,142
  • 8
  • 10
  • 7
    Nice, loooks like .INTERMEDIATE approach works fine. See also [Automake article](http://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html#Multiple-Outputs) describing the problem (but they try to solve it in a portable way). – Grwlf Jul 25 '13 at 13:36
  • 3
    This is the best answer I saw to this. Also of interest could be the other special targets: https://www.gnu.org/software/make/manual/html_node/Special-Targets.html – Arne Babenhauserheide May 21 '14 at 07:39
  • 3
    @deemer This is not working for me on OSX. (GNU Make 3.81) – Jorge Bucaran Dec 11 '15 at 05:35
  • 3
    I've been trying this, and I've noticed subtle problems with parallel builds --- dependencies don't always propagate properly across the intermediate target (although everything is fine with `-j1` builds). Also, if the makefile gets interrupted and it leaves the `input.in.intermediate` file around it gets really confused. – David Given Jun 15 '16 at 23:57
  • 6
    This answer has an error. Yo can have parallel builds working perfectly. You just have to change `file-a.out file-b.out: input.in.intermediate` to `file-a.out file-b.out: input.in.intermediate ;` See https://stackoverflow.com/questions/37873522/unreliable-parallel-builds-in-a-makefile-with-intermediate for more details. – siulkilulki Nov 30 '18 at 12:20
43

After GNU Make 4.3 (19th Jan. 2020) you can use “grouped explicit targets”. Replace : with &:.

file-a.out file-b.out &: input.in
    foo-bin input.in file-a.out file-b.out

The NEWS file of the GNU Make says:

New feature: Grouped explicit targets

Pattern rules have always had the ability to generate multiple targets with a single invocation of the recipe. It's now possible to declare that an explicit rule generates multiple targets with a single invocation. To use this, replace the ":" token with "&:" in the rule. To detect this feature search for 'grouped-target' in the .FEATURES special variable. Implementation contributed by Kaz Kylheku <kaz@kylheku.com>

Kazuki Okamoto
  • 661
  • 5
  • 10
11

I would solve it as follows :

file-a.out: input.in
    foo-bin input.in file-a.out file-b.out   

file-b.out: file-a.out
    #do nothing
    noop

In this case parallel make will 'serialize' creating a and b but since creating b does not do anything it takes no time.

Peter Tillemans
  • 34,983
  • 11
  • 83
  • 114
  • 18
    Actually, there is a problem with this. If `file-b.out` was created as stated and then mysteriously deleted, `make` will be unable to recreate it because `file-a.out` will still be present. Any thoughts? – makesaurus Jun 08 '10 at 11:06
  • If you mysteriously delete `file-a.out` then the above solution works well. This suggests a hack: when only one of a or b exist, then the dependencies should be ordered such that missing file appears as output of `input.in`. A few `$(widcard...)` s, `$(filter...)` s, `$(filter-out...)` s etc., should do the trick. Ugh. – bobbogo Jan 12 '11 at 20:38
  • 3
    P.S. `foo-bin` will obviously create one of the files first (using microsecond resolution), so you must ensure you have the correct dependency ordering when both files exist. – bobbogo Jan 12 '11 at 20:41
  • 2
    @bobbogo either that, or add `touch file-a.out` as a second command after the `foo-bin` invocation. – Connor Harris Apr 19 '17 at 12:48
  • Here is a potential fix for this: add `touch file-b.out` as a second command in the first rule and duplicate the commands in the second rule. If either file is missing or older than `input.in`, the command will be executed once only and will regenerate both files with `file-b.out` more recent than `file-a.out`. – chqrlie May 31 '19 at 17:56
10

This is based on @deemer's second answer which does not rely on pattern rules, and it fixes an issue I was experiencing with nested uses of the workaround.

file-a.out file-b.out: input.in.intermediate
    @# Empty recipe to propagate "newness" from the intermediate to final targets

.INTERMEDIATE: input.in.intermediate
input.in.intermediate: input.in
    foo-bin input.in file-a.out file-b.out

I would have added this as a comment to @deemer's answer, but I can't because I just created this account and don't have any reputation.

Explanation: The empty recipe is needed in order to allow Make to do the proper bookkeeping to mark file-a.out and file-b.out as having been rebuilt. If you have yet another intermediate target which depends on file-a.out, then Make will choose to not build the outer intermediate, claiming:

No recipe for 'file-a.out' and no prerequisites actually changed.
No need to remake target 'file-a.out'.
Richard Xia
  • 664
  • 7
  • 14
2

This is how I do it. First I always separate pre-requesits from the recipes. Then in this case a new target to do the recipe. 

all: file-a.out file-b.out #first rule

file-a.out file-b.out: input.in

file-a.out file-b.out: dummy-a-and-b.out

.SECONDARY:dummy-a-and-b.out
dummy-a-and-b.out:
    echo creating: file-a.out file-b.out
    touch file-a.out file-b.out

The first time:
1. We try to build file-a.out, but dummy-a-and-b.out needs doing first so make runs the dummy-a-and-b.out recipe.
2. We try to build file-b.out, dummy-a-and-b.out is up to date.

The second and subsequent time:
1. We try to build file-a.out: make looks at prerequisites, normal prerequisites are up to date, secondary prerequisites are missing so ignored.
2. We try to build file-b.out: make looks at prerequisites, normal prerequisites are up to date, secondary prerequisites are missing so ignored.

ctrl-alt-delor
  • 7,506
  • 5
  • 40
  • 52
0

As an extension to @deemer's answer, I have generalised it into a function.

sp :=
sp +=
inter = .inter.$(subst $(sp),_,$(subst /,_,$1))

ATOMIC=\
    $(eval s1=$(strip $1)) \
    $(eval target=$(call inter,$(s1))) \
    $(eval $(s1): $(target) ;) \
    $(eval .INTERMEDIATE: $(target) ) \
    $(target)

$(call ATOMIC, file-a.out file-b.out): input.in
    foo-bin input.in file-a.out file-b.out

Breakdown:

$(eval s1=$(strip $1))

Strip any leading/trailing whitespace from the first argument

$(eval target=$(call inter,$(s1)))

Create a variable target to a unique value to use as the intermediate target. For this case, the value will be .inter.file-a.out_file-b.out.

$(eval $(s1): $(target) ;)

Create an empty recipe for the outputs with the unique target as a depedency.

$(eval .INTERMEDIATE: $(target) )

Declare the unique target as an intermediate.

$(target)

Finish with a reference to the unique target so this function can be used directly in a recipe.

Also note the use of eval here is because eval expands to nothing so the full expansion of the function is just the unique target.

Must give credit to Atomic Rules in GNU Make which this function is inspired from.

Lewis R
  • 433
  • 5
  • 9
0

As of version 4.3, GNU make supports so-called "grouped targets":

  • New feature: Grouped explicit targets

    Pattern rules have always had the ability to generate multiple targets with a single invocation of the recipe. It's now possible to declare that an explicit rule generates multiple targets with a single invocation. To use this, replace the ":" token with "&:" in the rule. To detect this feature search for 'grouped-target' in the .FEATURES special variable.

    Implementation contributed by Kaz Kylheku address@hidden

Release notes: https://lists.gnu.org/archive/html/make-w32/2020-01/msg00001.html

Documentation: https://www.gnu.org/software/make/manual/make.html#Multiple-Targets

TL;DR: use &: instead of :.

jonas-schulze
  • 238
  • 1
  • 13
-1

To prevent parallel multiple execution of a rule with multiple outputs with make -jN, use .NOTPARALLEL: outputs. In your case :

.NOTPARALLEL: file-a.out file-b.out

file-a.out file-b.out: input.in
    foo-bin input.in file-a.out file-b.out
guilloptero
  • 1,583
  • 1
  • 10
  • 6