4

I have a nonrecursive build system, where parsing happens once. While parsing, I create a dependency chain for binary <- library <- object

The same source code is compiled for multiple hardware variants.

There is target all which depends on variant/all. There are variants/variant_name/all for each hardware needed by variant/all. variant/variant_name/all depends on all the binaries.

As parsing would have constructed the rest of the dependency, a simple make all does the job for me.

The requirement now is: We have more than one customer, whose toolchain is different for the same hardware variant. How can I parse once, and run build for all customers at once putting Customer Specific binaries, libraries etc into their own directories.

The following are the options I thought about.

  1. Parse multiple times(== number of customers), and call make multiple times. Split variant/all into two(==number of customers) all_customer1, all_customer2 variant/all depends on all_customer1 all_customer2 This is working almost fine! But, I'm interested in parsing once, and serving the purpose.

  2. Target specific variables feature of Make. But, this cannot be used in my case because I don't know all the targets already, and so the the dependency graph is not predefined and so I cannot really go with this feature.

Our system is perhaps more complex than I explained. But, from the comments I hear, I'm hoping to get some more help. Appreciate your time.

venkrao
  • 383
  • 4
  • 17
  • 1
    Can you explain more clearly what you mean when you say target specific variables won't work? And why you think splitting the customer targets requires parsing more than once and calling make more than once? If you split the targets then they should be able to each have their own variables which should still let you run make once, no? – Etan Reisner Dec 03 '13 at 13:56
  • Because the dependency graph for binary <- library <- object is constructed dynamically. The path to these target files is constructed dynamically while parsing once. I'm unable to figure out a way where the "target" file itself is initialized via target-specific variable. As in, I must write something like binary:$(variable) := customer But, I don't really have the list of object files and libraries already even before parsing starts. And thats the bigger challenge I'm seeing. – venkrao Dec 03 '13 at 15:56
  • Furthermore, as I thought already, it looks pretty tough to influence target specific variables for the dependencies. http://stackoverflow.com/questions/1340060/target-specific-variables-as-prerequisites-in-a-makefile However, there seems to be some other approaches to this. I'm reading more right on .. – venkrao Dec 03 '13 at 16:08
  • Why would you need a target specific variable as a prerequisite? Are you building your toolchain too? I was assuming you were just needing to override things like $(CC), $(AR), etc. that represent existing tools. – Etan Reisner Dec 03 '13 at 17:58
  • You don't need the target specific variables on the binaries if you can set them on a phony target "above" the binaries in the dependency chain. So if you have a `variant/hw1_customer1/all` target you could set the specific variable there and then when building dependencies of that target they would override default variable values, etc. – Etan Reisner Dec 03 '13 at 17:59
  • Etan, what if my build is run only for the object file? The phony target needs the binary, library, and object to be built. But, when I just build the object? Then what happens? I'm pretty unclear. I see it difficult to simulate my build environment, and I'm tight targets. So, I have split the makefiles, and used customer specific variable names in the main makefile, and the customer specific files point to the main file via softlink. This is working just fine for me. Perhaps my explanation is not so clear. Essentially, I have a mix of both target specific variables, softlinks to makefile. – venkrao Dec 08 '13 at 18:20

1 Answers1

2

make inherits target-specific variables, so long as they have a unique path, so the following will work:

.PHONY: all
all: all-customer1 all-customer2

.PHONY: all-customer1
all-customer1: CFLAGS=-ggdb3 -O0
all-customer1: $(outdir)customer1/bin/prog

.PHONY: all-customer2
all-customer2: CFLAGS=-O3
all-customer2: $(outdir)customer2/bin/prog

$(outdir)customer1/bin/prog: $(outdir)customer1/src/main.o
    $(CC) -o $@ $(CFLAGS) $<

$(outdir)customer2/bin/prog: $(outdir)customer2/src/main.o
    $(CC) -o $@ $(CFLAGS) $<

$(outdir)customer1/src/main.o: src/main.c
    $(CC) -c -o $@ $(CFLAGS) $^

$(outdir)customer2/src/main.o: src/main.c
    $(CC) -c -o $@ $(CFLAGS) $^

meaning that every top-level target gets built using a consistent set of flags, whereas the following doesn't:

.PHONY: all
all: all-customer1 all-customer2

.PHONY: all-customer1
all-customer1: CFLAGS=-ggdb3 -O0
all-customer1: $(outdir)customer1/bin/prog

.PHONY: all-customer2
all-customer2: CFLAGS=-O3
all-customer2: $(outdir)customer2/bin/prog

$(outdir)customer1/bin/prog: src/main.o
    $(CC) -o $@ $(CFLAGS) $<

$(outdir)customer2/bin/prog: src/main.o
    $(CC) -o $@ $(CFLAGS) $<

src/main.o: src/main.c
    $(CC) -c -o $@ $(CFLAGS) $^

because you end up src/main.o only being compiled once, which is not what you want. So, fundamentally, you must create different physical targets for each toolchain/set of options.

With respect to parsing, how long does it take vs. running the actual build? My experience is that even a very large code base can be parsed into a single heirarchical build tree in a tiny fraction of the time it takes to actually execute the build (even with monsterous build servers...).

So, with these two facts in mind, are you sure that this is what you want to do? Given that you are basically running several builds, surely it makes sense to do just that, especially as it would make it much easier to scale; if you have a new customer, you just add a new build job and if running all those builds takes a long time, you can run them on different servers, e.g. Jenkins slaves.

Sure, it's not as 'clean' and does leave open the posibility of it 'working for some, but not for others', but that would always be a management issue anyway...

If you want to support multiple customer profiles in a nice way, you can do it by using a configuration file which is setup something like this:

# Build config for "Customer X"
# Costomer identifier: customer1
customer1_cflags=-O0 -ggdb3

.PHONY: all-customer1
all-customer1: CFLAGS=$(customer1_cflags)
all-customer1: all

And then, in your Makefile:

# While convenient, Wildcards are harmful - they make your builds more fragile.
# This, for example, will fail fi there are no configs...
-include $(wildcard configs/*.mk)

This will then enable you to run either make all to get your stock/reference build or make all-customer1 to run a customised build for your customer - if you've used a variable for specifying your working/output directory, then you could override that on the command line:

make outdir=customer1_build/ all-customer1

Note: variable overrides are based on 'closest' override - if someone has overriden a variable on a rule directly, then this approach will breakdown.

To go beyond this, to create "One Build Tree to Rule them All", you'd need to build a list of sources, then dynamically create the targets using eval. The example below demonstrates the principle, but would need some extra work to make production ready.

Edit: I've included comments between the define ... endef, but these would need to be removed before the code will function correctly.

The main Makefile:

BUILD=build/
outdir=$(or $(filter %/,$(strip $(BUILD))),$(strip $(BUILD))/)

program=bin/hello

include platforms/all.mk
include configs/all.mk

# This creates all the necessary constructs for building a
# config/platform combination.
define CreateBuildConfig =
all: all-$(1)-$(2)

.PHONY: all-$(1)-$(2)
all-$(1)-$(2): $(outdir)$(1)/$(2)/$(3)

# Create implicit rule for building sources to objects
$(outdir)$(1)/$(2)/%.o: %.c
    $$(CC) -c -o $$@ $$(CFLAGS) $$(DEFINES) $$<

# Set the variables for this config/platform combo...
$(outdir)$(1)/$(2)/$(3): DEFINES=$$(foreach def,$$(platform_$(2)_defines),-D$$(def))
$(outdir)$(1)/$(2)/$(3): CFLAGS=$$(config_$(1)_cflags)

# The rule for creating the executable.
$(outdir)$(1)/$(2)/$(3): $(foreach obj,$(4:.c=.o),$(outdir)$(1)/$(2)/$(obj))
    $$(CC) -o $$@ $$(CFLAGS) $$(DEFINES) $$<

# Cleanup after ourselves.
clean: clean-$(1)-$(2)

clean-$(1)-$(2):
    -rm -f $(outdir)$(1)/$(2)/$(3)
    find $(outdir)$(1)/$(2) -name '*.o' -delete
endef

# Some top-level targets, for documentation purposes.
.PHONY: all
all:

.PHONY: clean
clean:

# Build the list of sources.
sources+=src/main.c

# Create the actual build targets, for each platform/config pair.
$(foreach platform,$(all_platforms),$(foreach config,$(all_configs),$(eval $(call CreateBuildConfig,$(config),$(platform),$(program),$(sources)))))

The platforms config file platforms/all.mk:

all_platforms:=model1 model2

include platforms/model1.mk
include platforms/model2.mk

Example platform file platforms/model1.mk:

_platform=model2

platform_$(_platform)_defines:=NAME=$(_platform)
platform_$(_platform)_defines+=MAX_FOO=16

_platform=

The configurations config file config/all.mk:

all_configs:=gcc48 customer1 customer2

include configs/gcc48.mk
include configs/customer1.mk
include configs/customer2.mk

Example config file customer1.mk:

_config=customer1

config_$(_config)_cflags=-O3

_config=
The Real Edward Cullen
  • 428
  • 2
  • 11