How to make a makefile for C and C++, with sources in subdirectories

Question

I have a project that presently builds in Xcode on OS X. I'm trying to construct a makefile to allow it to build on other Un*x systems. I'm a novice at writing makefiles, so I have been cobbling together a makefile from various examples on the web, and perhaps unsurprisingly, I can't get the result to work. I've looked at other similar questions on SO and gleaned what I can from them, but my situation seems to be a bit different because

1. I have a large number of source files, so specifying all sources and objects explicitly is not practical, I need to use wildcards
2. Some of my sources are in subdirectories, and the same filename can be used in different subdirectories so objects need to be placed by their source files within the file hierarchy to avoid collisions
3. I'm building both C and C++ and linking it together at the end
4. I want to be able to build two different targets, named "slim" and "eidos", using different subsets of the source files in the directory.

So please don't mark this as a dup unless the other question really addresses all of these requirements; thanks.

I would also like to avoid the use of makefile-generation tools, partly because it adds another level of complexity that I don't presently understand, and partly because it seems like overkill since I'm not interested in handling header dependencies and such (see discussion below), and partly because I need maximum portability (I don't want to depend on particular tools being present beyond vanilla make).

The error I'm getting from make -n is a simple one, so my mistake is probably very dumb:

make: *** No rule to make target `gsl/complex/*.c.o', needed by `slim'.  Stop.

But it seems to me that *.c.o targets ought to be handled by my %.c.o rule, so I'm puzzled. My makefile:

SHELL = /bin/sh
CC = gcc
CXX = g++
INCLUDES = -iquote./eidos -iquote./gsl -iquote./gsl/rng -iquote./gsl/randist -iquote./gsl/sys -iquote./gsl/specfunc -iquote./gsl/complex
CCFLAGS = -O3 -v $(INCLUDES)
CXXFLAGS = -O3 -v $(INCLUDES) -std=c++11

OUTPUTDIR = ./bin/
MKDIR = mkdir -p $(OUTPUTDIR)

CSOURCES = ./gsl/*/*.c
SLIM_CXXSOURCES = ./core/*.cpp ./eidos/*.cpp
EIDOS_CXXSOURCES = ./eidostool/*.cpp ./eidos/*.cpp

COBJECTS = $(patsubst %.c, %.c.o, $(CSOURCES))
SLIM_CXXOBJECTS = $(patsubst %.cpp, %.cpp.o, $(SLIM_CXXSOURCES))
EIDOS_CXXOBJECTS = $(patsubst %.cpp, %.cpp.o, $(EIDOS_CXXSOURCES))

all: slim eidos FORCE

slim: $(COBJECTS) $(SLIM_CXXOBJECTS) FORCE
    $(MKDIR)
    $(CXX) $(COBJECTS) $(SLIM_CXXOBJECTS) -o ./bin/slim

eidos: $(COBJECTS) $(EIDOS_CXXOBJECTS) FORCE
    $(MKDIR)
    $(CXX) $(COBJECTS) $(EIDOS_CXXOBJECTS) -o ./bin/eidos

%.cpp.o : %.cpp
    $(CXX) -c $(CXXFLAGS) -o $@ $<

%.c.o : %.c
    $(CC) -c $(CCFLAGS) -o $@ $<

clean: FORCE
    $(RM) -rf $(OUTPUTDIR)
    $(RM) ./*.o

# Would use .PHONY, but we don't want to depend on GNU make.
# See http://www.gnu.org/software/make/manual/make.html#Phony-Targets
# and http://www.gnu.org/software/make/manual/make.html#Force-Targets
FORCE:

I'm not trying to establish per-file header dependencies or anything smart like that. Instead, I'm just trying to use FORCE to force a full rebuild each time one of the top-level targets (slim, eidos, all) is built. This is because I don't expect people to use this makefile during development; all development is done on OS X in Xcode. So it doesn't need to be minimal/efficient, it just needs to work reliably to produce a final product. I'm not sure that I'm using FORCE correctly, though.

I also have a couple of side questions.

1. I took the $< endings for the two compilation commands from example makefiles on the web, but I don't know what they do. What's going on there? It looks like a redirection, but it's not a redirection command I'm familiar with, and of course Google is useless for searching for symbolic things like this.
2. The patsubst business is a bit tricky, and I don't know how to tell whether it's working or not. I tried make -np to print out make's internal info, but it seems to show variables with their original definitions, not with their resolved definitions, so I can't tell what the final value is that make will actually use for my variables. How can I debug this? And is patsubst a standard part of make? Is there a better way to do what I'm trying to do there?

Thanks. Sorry for the hectic multi-part question. By the way, all I'm really trying to do is convert the previously existing makefile to build source files individually, producing .o object files, and then link at the end. The previously existing makefile built everything in a single call to g++, which is causing problems for some users because of excessive memory usage and build time. Here's the old Makefile:

SHELL = /bin/sh
CC = g++
CFLAGS = -O3 -Wno-deprecated-register
INCLUDES = -iquote./eidos -iquote./gsl -iquote./gsl/rng -iquote./gsl/randist -iquote./gsl/sys -iquote./gsl/specfunc -iquote./gsl/complex
ALL_CFLAGS = $(CFLAGS) $(INCLUDES) -std=c++11

all: slim eidos FORCE

slim: FORCE
    mkdir -p ./bin
    $(CC) $(ALL_CFLAGS) ./core/*.cpp ./eidos/*.cpp ./gsl/*/*.c -o ./bin/slim

eidos: FORCE
    mkdir -p ./bin
    $(CC) $(ALL_CFLAGS) ./eidostool/*.cpp ./eidos/*.cpp ./gsl/*/*.c -o ./bin/eidos

clean: FORCE
    rm -f ./bin/slim
    rm -f ./bin/eidos

# Would use .PHONY, but we don't want to depend on GNU make.
# See http://www.gnu.org/software/make/manual/make.html#Phony-Targets
# and http://www.gnu.org/software/make/manual/make.html#Force-Targets
FORCE:

That works fine, apart from the aforementioned issue of being slow and using a ton of memory.

Answer 1

The basic concept of make is actually quite simple: for each file you want to be able to build, you specify what its prerequisites are, and how to construct the target file from those prerequisites. Make can figure out how to chain together multiple build steps to create the ultimate target you request from the resources actually present, if in fact there as any way to do so.

Although GNU make is indeed powerful, and has been widely ported, I generally recommend avoiding extensions specific to that version of make. Just because you can get GNU make for just about any machine does not alleviate confusion when that is not the default make on a given machine, nor does it stave off consternation if one actually has to obtain and install GNU make before being able to build your software. And your particular project has a simple enough structure (number of source files notwithstanding) that you don't need GNU extensions.

With respect to your qualifications:

I have a large number of source files, so specifying all sources and objects explicitly is not practical, I need to use wildcards

In comments you clarified that "a large number of source files" means "more than 100", which I guess means that "large" is in the eye of the beholder. What you're describing is certainly not small, but it's nevertheless fairly modest. I've built make-based build systems for software suites with many hundreds of source files.

Some of my sources are in subdirectories, and the same filename can be used in different subdirectories so objects need to be placed by their source files within the file hierarchy to avoid collisions

This is not a problem at all. Make itself doesn't really know about directories; it works mainly with strings, and lets the shell commands it executes interpret those as appropriate. There is no special problem with target and prerequiste names that contain relative or even absolute paths.

I'm building both C and C++ and linking it together at the end

Again, not a big problem, or at least not one specific to make. The problem here is that how you link C to C++ may be dependent on the underlying toolchain (compilers and linker).

I want to be able to build two different targets, named "slim" and "eidos", using different subsets of the source files in the directory

No problem at all. This is very common.

You also remark that

I'm just trying to use FORCE to force a full rebuild each time one of the top-level targets (slim, eidos, all) is built. This is because I don't expect people to use this makefile during development; all development is done on OS X in Xcode. So it doesn't need to be minimal/efficient, it just needs to work reliably to produce a final product. I'm not sure that I'm using FORCE correctly, though.

I'm sorry, but that just doesn't make sense. If people aren't using the makefile during development, then the expected use case is that nothing has yet been built, so not forcing is needed. On the other hand, what is gained by forcing? Not much. The only reason to force is that something about your project or make file makes it impossible for make to correctly evaluate which targets need to be rebuilt. In any event, if you provide a proper clean target then the user can easily get a clean build whenever that's what they want. Invest power in your users. It pays off.

I took the $< endings for the two compilation commands from example makefiles on the web, but I don't know what they do. What's going on there?

Make provides a few automatic variables; one of them is named <. In a build rule, its value, accessed via the syntax $<, expands to the name of the first prerequisite of the current rule.

The patsubst business is a bit tricky, and I don't know how to tell whether it's working or not.

You don't need it, but you can test whether it's working by including a command in one of your build rules that echos its result.

Here's a template for a makefile that would probably work for you:

CC = gcc
CXX = g++

INCLUDES = -Ieidos -Igsl -Igsl/rng -Igsl/randist -Igsl/sys -Igsl/specfunc -Igsl/complex
CFLAGS = -O3 -Wno-deprecated-register
CXXFLAGS = $(CFLAGS) -std=c++11

CSOURCES = \
    gsl/subdir/something.c \
    gsl/subdir/something_else.c \
    gsl/dir2/important.c

COMMON_CXXSOURCES = \
    eidos/file1.cpp \
    eidos/file2.cpp

SLIM_CXXSOURCES = \
    core/slim1.cpp \
    core/slim2.cpp \
    core/slim3.cpp

EIDOS_CXXSOURCES = \
    eidostool/et.cpp \
    eidostool/et42.cpp \
    eidostool/phone_home.cpp

COMMON_OBJS = $(CSOURCES:.c=.o) $(COMMON_CXXSOURCES:.cpp=.o)
SLIM_OBJS = $(SLIM_CXXSOURCES:.cpp=.o)
EIDOS_OBJS = $(EIDOS_CXXSOURCES:.cpp=.o)

all: slim eidos

# NOTE: commands in build rules must start with a tab character, which
# will not be conveyed via the web representation of what follows.

# How to link object files together to build slim, using the C++ compiler as
# the linker driver
slim: $(SLIM_OBJS) $(COMMON_OBJS)
    $(CXX) $(CXXFLAGS) -o $@ $^

# How to link object files together to build eidos, using the C++ compiler as
# the linker driver
eidos: $(EIDOS_OBJS) $(COMMON_OBJS)
    $(CXX) $(CXXFLAGS) -o $@ $^

# remove all built files
clean:
    rm -f slim eidos $(COMMON_OBJS) $(SLIM_OBJS) $(EIDOS_OBJS)

# You might not actually need anything below, because `make` has built-in
# rules for building object files from source files of various kinds, based
# on their extensions.

# How to build an object file (x.o) from a corresponding C source file (x.c)
.c.o:
    $(CC) $(INCLUDES) $(CFLAGS) -c -o $@ $<

# How to build an object file from a corresponding C++ source file (x.cpp)
.cpp.o:
    $(CXX) $(INCLUDES) $(CXXFLAGS) -c -o $@ $<

Note about wildcards: I recommend not using them, and instead specifying explicitly which sources need to be included in the build. Yes, that becomes something you have to maintain, but it also allows you to have source files in your tree that don't contribute to the build. These could be backup copies, working copies, etc.. And as long as you test building the project via make before you release it, it should be easy to catch omissions.

Answer 2

Not entirely sure this will work for you, there may be some surprises in your setup that are not immediately obvious.

But this should give you an idea of how you can build up a Makefile that selects different sets of source/objects using GNU Make.

CC = gcc
CXX = g++
INCLUDES = -iquote./eidos -iquote./gsl -iquote./gsl/rng -iquote./gsl/randist -iquote./gsl/sys -iquote./gsl/specfunc -iquote./gsl/complex
CCFLAGS = -O3 
CXXFLAGS = -O3 -std=c++11
CPPFLAGS = $(INCLUDES)

OUTPUTDIR = ./bin/
MKDIR = mkdir -p $(OUTPUTDIR)

# shell find command is good for multiple level
# directory searching
CSOURCES := $(shell find gsl -name "*.c")

# wildcard selects all matching files
SLIM_SOURCES = \
    $(CSOURCES) \
    $(wildcard core/*.cpp) \
    $(wildcard eidos/*.cpp)

# filter selects one type of file for substitution
# I use it here to prevent source files that don't match the 
# pattern from being included
SLIM_OBJECTS := \
    $(patsubst %.c,%.c.o,$(filter %.c,$(SLIM_SOURCES))) \
    $(patsubst %.cpp,%.cpp.o,$(filter %.cpp,$(SLIM_SOURCES)))

EIDOS_SOURCES = \
    $(CSOURCES) \
    $(wildcard eidostool/*.cpp) \
    $(wildcard eidos/*.cpp)

EIDOS_OBJECTS := \
    $(patsubst %.c,%.c.o,$(filter %.c,$(EIDOS_SOURCES))) \
    $(patsubst %.cpp,%.cpp.o,$(filter %.cpp,$(EIDOS_SOURCES)))

all: $(OUTPUTDIR)/slim $(OUTPUTDIR)/eidos

# I often find a show target useful for complex makefiles
# to figure out if you are setting the variables correctly
show:
    @echo "SLIM_SOURCES: $(SLIM_SOURCES)"
    @echo "SLIM_OBJECTS: $(SLIM_OBJECTS)"
    @echo "EIDOS_SOURCES: $(EIDOS_SOURCES)"
    @echo "EIDOS_OBJECTS: $(EIDOS_OBJECTS)"

$(OUTPUTDIR)/slim: $(SLIM_OBJECTS)
    $(CXX) -o $@ $^ $(LDFLAGS)

$(OUTPUTDIR)/eidos: $(EIDOS_OBJECTS)
    $(CXX) -o $@ $^ $(LDFLAGS)

%.cpp.o : %.cpp
    $(CXX) -c $(CXXFLAGS) $(CPPFLAGS) -o $@ $<

%.c.o : %.c
    $(CC) -c $(CCFLAGS) $(CPPFLAGS) -o $@ $<

clean: 
    $(RM) $(OUTPUTDIR)/slim $(OUTPUTDIR)/eidos $(SLIM_OBJECTS) $(EIDOS_OBJECTS)