Makefile - compile multiple C file at once

Question

This question is different from the one at makefiles - compile all c files at once in the sense that I have one extra requirement: I want to redirect all the object files in a separate directory.

Here is the setup:

I have multiple sources in a directory say src/mylib.
I want the objects files to end up in build/mylib.
Please note also that under mylib there are subdirectories.

The first attempt was as follows:

sources = $(shell find src/ -name ".c")
objects_dirs = $(subst src/, build/, $(dir $(sources)) # This variable is used by the build rule to create directories for objects files prior to compilation
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory

# This is where things aren't working as expected
$(objects): build $(sources)
    $(cc) $(cflags) -o $@ $(word 2, $^))

build:
    $(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))

For the makefile above, only one object file was being generated. I guessed this might have something to do with GCC only being able to generate one object file at a time. Regardless of that, checking the values of $@ and $(word 2, $^) in the $(objects) target shows that only one file is being considered even though I have multiple files.

So I changed my makefile to the following:

sources = $(shell find src/ -name ".c")
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory

# This works as expected but it appears to me like make is generating all the objects files even though source files did not change. This can be seen by checking the timestamps on new object files after running make again.
$(objects): build $(sources)
    $(foreach source, $(sources), $(shell $(cc) $(cflags) -o $(subst src/,build/, $(patsubst %.o,%.c,$(source))) $(source)))

build:
    $(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))

The second makefile works as expected but objects files are being rebuilt again which defeats another purpose of using make: only recompile those source files that changed from the last compilation.

Hence my question: how does one generate all object files in a separate directory at once (by this I mean perform the compilation of all sources files in one rule) while making sure that if a source file didn't change the associated object file should not be regenerated.

I am not after speeding up compilation. What I seek is one rule that will generate all objects files such that only updated source files should be recompiled.

The last makefile does the job but there is a recompiling of all source files which defeats another purpose of using make: only changed source files should be recompiled.

EDIT

After reading comments, it appears I have not phrased my question properly. As the details of what I have are already present, I leave the question as it is with additional details below.

The second makefile in the source code above does work. But it does only half the job. The build directory effectively mirrors the src directory.
So if I have say a file as src/mylib/point/point.c, I get build/mylib/point/point.o generated. This is the first part.
The second part is that if point.c does not changes, point.o in the build/mylib/point/ directory must not be regenerated. But after checking timestamps on the object file, I can tell that a new object file replaced the old one after running make again. This is not good because for large projects, compilation time remains O(n) with n being the number of source files to compile.

So this question is about how to preserve the second makefile without make regenerating object files.
From what I can gather from comments, I am asking too much from make. But if anyone knows how to make this happen, I leave the question open.

Answer 1

Makefile:

all:
clean:

src_root := src
src_subdirs := foo foo/bar foo/bar/buz
build_root := build

o_suffix := .o

# Build list of sources. Iterate every subfolder from $(src_subdirs) list 
# and fetch all existing files with suffixes matching the list.
source_suffixes := .c .cpp .cxx
sources := $(foreach d,$(addprefix $(src_root)/,$(src_subdirs)),$(wildcard $(addprefix $d/*,$(source_suffixes))))

# If src_subdirs make variable is unset, use 'find' command to build list of sources.
# Note that we use the same list of suffixes but tweak them for use with 'find'
ifeq ($(src_subdirs),)
  sources := $(shell find $(src_root) -type f $(foreach s,$(source_suffixes),$(if $(findstring $s,$(firstword $(source_suffixes))),,-o) -name '*$s'))
endif

$(info sources=$(sources))

# Build source -> object file mapping.
# We want map $(src_root) -> $(build_root) and copy directory structure 
# of source tree but populated with object files.
objects := $(addsuffix $(o_suffix),$(basename $(patsubst $(src_root)%,$(build_root)%,$(sources))))
$(info objects=$(objects))

# Generate rules for every .o file to depend exactly on corresponding source file.
$(foreach s,$(sources),$(foreach o,$(filter %$(basename $(notdir $s)).o,$(objects)),$(info New rule: $o: $s)$(eval $o: $s)))

# This is how we compile sources:
# First check if directory for the target file exists. 
# If it doesn't run 'mkdir' command.
$(objects): ; $(if $(wildcard $(@D)),,mkdir -p $(@D) &&) g++ -c $< -o $@

# Compile all sources.
all: $(objects)
clean: ; rm -rf $(build_root)

.PHONY: clean all

Environment:

$ find
.
./src
./src/foo
./src/foo/bar
./src/foo/bar/bar.cxx
./src/foo/bar/buz
./src/foo/bar/buz/buz.c
./src/foo/bar/foo.c
./src/foo/foo.cpp

Run makefile:

$ make -f /cygdrive/c/stackoverflow/Makefile.sample -j
sources=src/foo/bar/bar.cxx src/foo/bar/buz/buz.c src/foo/bar/foo.c src/foo/foo.cpp
objects=build/foo/bar/bar.o build/foo/bar/buz/buz.o build/foo/bar/foo.o build/foo/foo.o
New rule: build/foo/bar/bar.o: src/foo/bar/bar.cxx
New rule: build/foo/bar/buz/buz.o: src/foo/bar/buz/buz.c
New rule: build/foo/bar/foo.o: src/foo/bar/foo.c
New rule: build/foo/foo.o: src/foo/bar/foo.c
New rule: build/foo/bar/foo.o: src/foo/foo.cpp
New rule: build/foo/foo.o: src/foo/foo.cpp
mkdir -p build/foo/bar && g++ -c src/foo/bar/bar.cxx -o build/foo/bar/bar.o
mkdir -p build/foo/bar/buz && g++ -c src/foo/bar/buz/buz.c -o build/foo/bar/buz/buz.o
mkdir -p build/foo/bar && g++ -c src/foo/bar/foo.c -o build/foo/bar/foo.o
mkdir -p build/foo && g++ -c src/foo/bar/foo.c -o build/foo/foo.o

Environment again:

$ find
.
./build
./build/foo
./build/foo/bar
./build/foo/bar/bar.o
./build/foo/bar/buz
./build/foo/bar/buz/buz.o
./build/foo/bar/foo.o
./build/foo/foo.o
./src
./src/foo
./src/foo/bar
./src/foo/bar/bar.cxx
./src/foo/bar/buz
./src/foo/bar/buz/buz.c
./src/foo/bar/foo.c
./src/foo/foo.cpp

Try running this Makefile with 'src_subdirs=' to exercise another approach to locate sources. Output should be the same.

Answer 2

I finally had some time to experiment with this, so here is what I came up with:

BUILD_DIR = build
SRC_DIR = src
SOURCES = $(shell find $(SRC_DIR)/ -name "*.c")
TARGET  = program
OBJECTS = $(SOURCES:$(SRC_DIR)/%.c=$(BUILD_DIR)/%.o)

default: $(TARGET)

.SECONDEXPANSION:

$(OBJECTS) : $$(patsubst $(BUILD_DIR)/%.o,$(SRC_DIR)/%.c,$$@)
        mkdir -p $(@D)
        $(CC) -c -o $@ $(CFLAGS) $<

$(TARGET): $(OBJECTS)
        $(CC) -o $@ $(CFLAGS) $^

.PHONY: default

Points of interest:

• I had to change the sources find pattern from ".c" to "*.c", I'm not sure if it depends on the exact shell used, but if you want to stay portable, be sure to use a widely accepted pattern.

• The .SECONDEXPANSION: is needed to enable the $$ rules for GNU Make. It is needed to allow target based substitution rules in the prerequisites for the $(OBJECTS).

• The prerequisite $$(patsubst $(BUILD_DIR)/%.o,$(SRC_DIR)/%.c,$$@) is saying, that the current target depends on a specific source file with the same folder structure and name.

• The command mkdir -p $(@D) is ensuring, that the path of the current target is created if it's missing.

Answer 3

If all you want is a single rule to handle all object files, without necessarily needing to "compile all at once" then you could have something like this:

BUILD_DIR = build
SOURCES = ...
TARGET  = ...
OBJECTS = $(SOURCES:%.c=$(BUILD_DIR)/%.o)

default: target

target: $(TARGET)

$(TARGET): $(OBJECTS)
    $(LD) -o $@ $(LDFLAGS) $^ $(LIBS)

$(BUILD_DIR)/%.o: %.c
    $(CC) -c -o $@ $< $(CFLAGS)

$(BUILD_DIR):
    -mkdir $@

_{[Note: This is written from memory and without testing.]}

Answer 4

After reading the GNU make manual again, here is a solution that solves the second problem.

The first attempt was the correct path. And the second attempt has the $(sources) in the prerequisites but does not use it in the commands and this is silly.

So the working makefile follows. It puts object files in a separate directory and it only compiles files that have changed.

sources = $(shell find src/ -name ".c")
$objects_dirs = $(subst src/, build/, $(dir $(sources)) # This variable is used by the build rule to create directories for objects files prior to compilation
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory

# This should now work as expected: object files go into their designated directories under "build/" and only updated files will be recompiled.
$(objects): build $(sources)
# After running say "make clean", make will figure out the need to run the first prerequisite.
# If we are doing a clean build, the number of prerequisites will equal the number of new prerequisites.
ifeq ($(words $?), $(words $^))
    # Note the use of "$?" instead of "$^". $? is used since it holds prerequisites that are newer than the target while $^ will holds all prerequisites whether they are new or not.
    $(foreach source, $(wordlist 2, $(words $?), $?), $(shell $(cc) $(cflags) -o $(subst src/,build, $(patsubst %.c,%.o, $(source))) $(source)))
else
    # If we have a few new targets, no need to exclude "build" from prerequisites because the first prerequisite will be a file that changed.
    $(foreach source, $?, $(shell $(cc) $(cflags) -o $(subst src/,build, $(patsubst %.c,%.o, $(source))) $(source)))
endif

.PHONY: build
build:
    $(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))

.PHONY: clean
clean:
    @rm -rf build/

The makefile is heavily commented with changes that made it work. The most important changes were:

• Use of $(foreach) to compile each file individually as required by GCC
• Use of $? to work only with prerequisites that are newer than the target
• Use of conditional to detected whether the first prerequisite has changed depending on circumstances. If we have a clean build (running make for the first time or after running make clean), the number of updated prerequisites will be the same as the number of newer prerequisites compared to the target. In other words $(words $?) == $(words $^) will be true. So we use this fact to exclude the firs prerequisite listed (build in our case) from the list of files to pass to GCC.

Also, when building the executable from the objects files, make sure to use $^ and not $? when selecting prerequisites else you will end up with only newer files in the executable and it will not run.

target = bin/mylib.a

.PHONY: all
all: $(target)

$(target): $(objects)
    ar -cvq $@ $^ # Notice that we're not using $? else only updated object files will end up in the archive.