4

I have a series (dozens) of projects that consist of large amounts of content in git repositories. Each repository has a git submodule of a common toolkit. The toolkit contains libraries and scripts needed to process the content repositories and build a publishable result. All the repositories are pushed to a host that runs CI and publishes the results. The idea is to keep the repeated code to an absolute minimum and mostly have content in the repositories and rely on and the toolkit to put it all together the same way for every project.

Each project has a top level Makefile that typically only has a couple lines, for example:

STAGE = stage

include toolkit/Makefile

The stage variable has some info about what stage this particular is in which determine which formats get built. Pretty much everything else is handled by the 600 line Makefile in the toolkit. Building some of the output formats can require a long chain of dependencies: The process of a source might trigger a target rule, but to get to the target there might be 8–10 intermediate dependencies where various files get generated before the final target can be made.

I've run across a couple situations where I want to completely replace (not just extend) a specific target rule in just one project. The target gets triggered in the middle of a chain of dependencies but I want to do something completely different for that one step.

I've tried just replacing the target in the top level Makefile:

STAGE = stage

%-target.fmt:
    commands

include toolkit/Makefile

This is specifically documented not to be supported, but tantalizingly it works sometime of the time. I've tried changing the order of declaring the custom target and the include but that doesn't seem to significantly affect this. In case it matters, yes, the use of patterns in targets is important.

Sometimes it is useful to have a makefile that is mostly just like another makefile. You can often use the ‘include’ directive to include one in the other, and add more targets or variable definitions. However, it is invalid for two makefiles to give different recipes for the same target.

Interestingly if I put custom functions in the top level Makefile below the include I can override the functions from the toolkit such that $(call do_thing) will use my override:

STAGE = stage

include toolkit/Makefile

define do_thing
    commands
endef

However the same does not seem to be true for targets. I am aware of the two colon syntax, but I do not want to just extend an existing target with more dependencies, I want to replace the target entirely with a different way of generating the same file.

I've thought about using recursive calls to make as suggested in the documentation, but then the environment including helper functions that are extensively setup in the toolkit Makefile would not be available to any targets in the top level Makefile. This would be a show stopper.

Is there any way to make make make an exception for me? Can it be coerced into overriding targets? I'm using exclusively recent versions of GNU-Make and am not too concerned about portability. Failing that is there another conceptual way to accomplish the same ends?

¹ My brain hasn't had enough coffee today. In trying to open Stack Overflow to ask this question I typed makeoverflow.com into my browser and was confused why auto-completion wasn't kicking in.

ucMedia
  • 4,105
  • 4
  • 38
  • 46
Caleb
  • 5,084
  • 1
  • 46
  • 65
  • Am I getting it right, your toolkit also contain the `%-target.fmt` rule with some default commands you want to override? – valir Dec 30 '16 at 18:14
  • @valir Yes. Both the prerequisites and recipe are different but the rule name is the same and I want to completely override the one from the toolkit with the one in the top level. – Caleb Dec 30 '16 at 18:26

2 Answers2

1

Updated answer: If your recipe has dependencies, these cannot be overriden by default. Then $(eval) might save you like this:

In toolkit have a macro definition with your generic rule:

ifndef TARGET_FMT_COMMANDS
define TARGET_FMT_COMMANDS
    command1 # note this these commands should be prefixed with TAB character
    command2
endef
endif

define RULE_TEMPLATE
%-target.fmt: $(1)
    $$(call TARGET_FMT_COMMANDS)

endef

# define the default dependencies, notice the ?= assignment
TARGET_DEPS?=list dependencies here

# instantiate the template for the default case
$(eval $(call RULE_TEMPLATE,$(TARGET_DEPS)))

Then into the calling code, just define TARGET_FMT_COMMANDS and TARGET_DEPS before including the toolkit and this should do the trick.

(please forgive the names of the defines/variables, they are only an example)

Initial answer: Well, I'd write this in the toolkit:

define TARGET_FMT_COMMANDS
    command1 # note this these commands should be prefixed with TAB character
    command2
endef

%-target.fmt:
    $(call TARGET_FMT_COMMANDS)

The you could simply redefine TARGET_FMT_COMMANDS after include toolkit/Makefile

The trick is to systematically have the TAB character preced the commands inside the definition if not you get weird errors.

You can also give parameters to the definition, just as usual.

valir
  • 311
  • 2
  • 6
  • I thought about. His, but unfortunately going this route you give up the normal functions of make. This won't allow me to redefine the prerequisites and hence trigger other possible targets that need calling. It work work to trick the recipe into having different content but the target hasn't been overridden. – Caleb Dec 30 '16 at 19:23
  • Oh, there were no mention about the pre-requisites in the question. You're out-of-luck with these as GNU Make expands them right away when it encounters the rule definition, so if you use the some $(DEPS) then override DEPS later, then you'll miss the latter AFAICT. – valir Dec 30 '16 at 19:30
  • But wait, there might be a solution for you. Use $(eval) macro expansion. I'll edit my answer as it's longer than a comment, – valir Dec 30 '16 at 19:30
0

I ran into the same issue, how I ended up working around the problem that overriding a target does not override the prerequisites was to override the pre-requisites' rules as well to force some of them to be empty commands.

in toolkit/Makefile:

test: depend depend1 depend2
   @echo test toolkit
...

in Makefile:

include toolkit/Makefile

depend1 depend2: ;

test: depend
   @echo test

Notice how depend1 and depend2 now have empty targets, so the test target's command is overridden and the dependencies are effectively overridden as well.

Matt Palmerlee
  • 2,668
  • 2
  • 27
  • 28
  • My testing last year showed that this worked _some_ of the time, but it wasn't consistent (see my comment in the question). Something about the way make parses itself multiple times it may or may not parse things in the order to override the your target, it may end up doing the opposite. What triggers this can be as simple as expansion of other variables or functions in unrelated targets, so while this hack might get you by for a couple fixed targets, in anything complicated or dynamic (especially with .SECONDEXPANSION:) it will fall apart and do unexpected things on you. – Caleb Dec 28 '17 at 16:49
  • Thanks for the heads up, it's a shame Makefiles were not really intended to be used this way since it seems like this would be a common use-case. – Matt Palmerlee Dec 28 '17 at 18:36
  • Ya the way the make parser procedes actually introduces several deranged behaviors. For example I found if I copied my overrides done as you describe here both before _and_ after the include they hade a better chance of getting over-ridden because depending on how many passes it parsed itself in (perhaps determined by how many layers of variable expansion were present?) sometimes the prior sometimes the later one would actually make the override happen. But it was just too fruity to use in production that way. – Caleb Dec 28 '17 at 18:48
  • Not being able to change the number of jobs from inside the makefile is another whacky one. You can set the right variable and it parses it, then _reparses_ the makefile, but by that time even though the variable is set every target got assigned a job queue on the first pass. And don't even get me started on secondary expansion in prerequisite lists. – Caleb Dec 28 '17 at 18:49
  • Yeah, it seems like as long as you keep things simple, Make can be pretty useful, especially just to keep from having to type/remember commonly used commands to install dependencies and run tests. What language are you using make with and have you considered [alternatives](https://stackoverflow.com/questions/66800/promising-alternatives-to-make)? – Matt Palmerlee Dec 28 '17 at 23:27
  • I would disagree and say you don't have to keep it simple. In fact it stands up to very complex scenarios. What you can't do is improvise—you really have to use each feature for the job it was intended for. If you use the wrong paradigm for a task it will bite you, but otherwise it does what it does well. In fact I haven't found an alternative that can hold its own in the mixed environments I work in. I'm rarely building single language projects (for which often there _are_ better tools), I'm [cobbling together lots of disparate pieces](https://github.com/alerque/casile/blob/master/makefile). – Caleb Dec 29 '17 at 14:30