I need help trimming down dependencies in a makefile

Question

I have created a makefile for the generation of a simple web page. The idea behind the makefile is this:

• We're compiling one web page, index.html
• index.html requires a stylus css main.sty that must be compiled
• There are a number of examples used in the page
  • The code for example one lives in lib/examples/one
  • Each example contains three parts
    • The markup (a .jade template file)
    • Some code (a .coffee script file)
    • A description (a .md markdown file)
• The build script must render each example into a single html file
  • Jade, Pygments, and Markdown are used to generate three html files
  • An example.jade template is used to combine these into one example file
    • example.jade must be copied to the correct build example directory, because the template language can only do relative imports. So in order to import example/one/code.html, we must copy the template to example/one and have it include code.html.
  • When finished, each example x will have compiled to tbuild/examples/x.html
• The lib/index.jade template is moved to build (so that it can include the example files)
• Jade is then used to compile the index.jade template into html

This is a wee bit of a simplification, but it's easier to understand this way. The simplification is that there are actually two markup files (left.html and right.html) in each example, and that the code file is both run through pygments and used as a script, so both code.html and code.coffee need to make it into build.

Right now, the makefile looks like this:

LIB = lib
BUILD = build
LIBEX = $(LIB)/examples
BUILDEX = $(BUILD)/examples
EXAMPLES = $(addsuffix .html,$(addprefix $(BUILDEX)/,$(shell ls $(LIBEX) | grep -v '.jade')))

all: $(BUILD)/main.css index.html

index.html: $(BUILD)/index.jade $(EXAMPLES)
    jade < $< --path $< > $@

$(BUILD)/index.jade: $(LIB)/index.jade
    mkdir -p $(@D)
    cp $< $@

$(BUILD)/main.css: $(LIB)/main.sty
    mkdir -p $(@D)
    stylus -u nib < $< > $@

$(BUILDEX)/%.html: $(BUILDEX)/%/template.jade $(BUILDEX)/%/left.html $(BUILDEX)/%/right.html $(BUILDEX)/%/code.html $(BUILDEX)/%/code.coffee $(BUILDEX)/%/text.html
    jade < $< --path $< > $@

$(BUILDEX)/%/template.jade: $(LIBEX)/template.jade
    mkdir -p $(@D)
    cp $< $@

$(BUILDEX)/%/left.html: $(LIBEX)/%/left.jade
    jade < $< --path $< > $@

$(BUILDEX)/%/right.html: $(LIBEX)/%/right.jade
    jade < $< --path $< > $@

$(BUILDEX)/%/code.html: $(LIBEX)/%/code.coffee
    pygmentize -f html -o $@ $<

$(BUILDEX)/%/code.coffee: $(LIBEX)/%/code.coffee
    mkdir -p $(@D)
    cp $< $@

$(BUILDEX)/%/text.html: $(LIBEX)/%/text.md
    markdown < $< > $@

clean:
    rm index.html -f
    rm $(BUILD) -rf

This works, but the problem is that when I touch "lib/examples/intro/code.coffee" and re-run make, I get the following:

mkdir -p build/examples/intro
cp lib/examples/template.jade build/examples/intro/template.jade
jade < lib/examples/intro/left.jade --path lib/examples/intro/left.jade > build/examples/intro/left.html
jade < lib/examples/intro/right.jade --path lib/examples/intro/right.jade > build/examples/intro/right.html
pygmentize -f html -o build/examples/intro/code.html lib/examples/intro/code.coffee
mkdir -p build/examples/intro
cp lib/examples/intro/code.coffee build/examples/intro/code.coffee
markdown < lib/examples/intro/text.md > build/examples/intro/text.html
jade < build/examples/intro/template.jade --path build/examples/intro/template.jade > build/examples/intro.html
jade < build/index.jade --path build/index.jade > index.html
rm build/examples/intro/right.html build/examples/intro/code.coffee build/examples/intro/code.html build/examples/intro/left.html build/examples/intro/text.html build/examples/intro/template.jade

Which, as you'll notice, is way more than needs to be done to regenerate the example. What I was hoping for was something more like this:

mkdir -p build/examples/intro
pygmentize -f html -o build/examples/intro/code.html lib/examples/intro/code.coffee
cp lib/examples/intro/code.coffee build/examples/intro/code.coffee
jade < build/examples/intro/template.jade --path build/examples/intro/template.jade > build/examples/intro.html
jade < build/index.jade --path build/index.jade > index.html

In other words, what I'm asking is:

1. What do I need to do to make the makefile not rebuild too much when I change something small?
   • In the above example, left.html, right.html, and text.html are all rebuilt when I touch code.coffee. How can I prevent this?
2. Why does make put the rm command at the end of the output? This seems like it might be causing some re-building where unnecessary.

Thanks for reading all the way through this beast of a question! I know that my make-fu is lacking, so any tips as to how to clean up the makefile and reduce redundancy are more than welcome!

Answer 1

This build system is too large and complex to reproduce easily -- and I hate to post solutions I haven't tested -- but try adding this line:

.SECONDARY:

EDIT:
I haven't been able to reproduce the behavior you describe, but I can offer some pointers.

The .SECONDARY: is a rule; it can go anywhere in the makefile. Basically, if Make detects a chain of chain of implicit rules, A->B->C, where A is a file that exists and C is the target, it considers B an intermediate file and will delete it once the job is done. The .SECONDARY: rule blocks the deletion.

You can combine rules that have the same commands. This:

foo: bar
    do something $< $@

baz: quartz
    do something $< $@

quince: geef
    do something $< $@

can be rewritten as this:

foo: bar

baz: quartz

quince: geef

foo baz quince:
    do something $< $@

That will remove a lot of redundancy in your makefile, and perhaps make things clearer.

Answer 2

Like @Beta mentioned, you are running into troubles with intermediate files here.

Each file that is not mentioned explicitely in the Makefile, but rather inferred via a pattern rule (a rule with %), is intermediate and is deleted right after make ran. And then remade on the next run.

That's the core of your "problem": The left.html and right.html files are deleted right away. Beta already mentioned that declaring them as .SECONDARY fixes this problem, like the make manual tells you.

By declaring a target .SECONDARY without any prerequisites anywhere in the Makefile (really: anywhere), you declare all targets as secondary - so, no auto-deletes.

Answer 3

You could also use PHONY for targets that are not files to improve the performance. Like all and clean. See here for more information: What is the purpose of .PHONY in a makefile?