Parallel make generates code multiple times

Question

let's assume an easy project consisting of two static sources and three dynamic ones, which are generated. The makefile looks like this:

SRC_DIR = src
OUTPUT = output

TARGET = $(OUTPUT)/lib.so

GEN_SRCS = dynamic_1.c dynamic_2.c dynamic_3.c
SRCS = static_1.c static_2.c $(GEN_SRCS)

OBJS = $(SRCS:%.c=$(OUTPUT)/%.o)

all: $(TARGET)

$(TARGET): $(OBJS)
    gcc -shared -o $@ $^

$(OUTPUT)/%.o: $(SRC_DIR)/$*%.c | $(OUTPUT)
    @echo "Creating object $@ from $^"
    gcc -c -o $@ $<

$(SRC_DIR)/dynamic_%.c: definition_for_generator.xml
    @echo "---------------------------------------"
    @echo "Generating dynamic Cs"
    @echo "---------------------------------------"
    ./generate.sh $<

$(OUTPUT):
    mkdir -p $(OUTPUT)

.PHONY: clean
clean:
    rm -rf $(OUTPUT) $(SRC_DIR)/dynamic_1.c $(SRC_DIR)/dynamic_2.c $(SRC_DIR)/dynamic_3.c

While building using one core only, it builds just fine:

mkdir -p output
Creating object output/static_1.o from src/static_1.c
gcc -c -o output/static_1.o src/static_1.c
Creating object output/static_2.o from src/static_2.c
gcc -c -o output/static_2.o src/static_2.c
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
Creating object output/dynamic_1.o from src/dynamic_1.c
gcc -c -o output/dynamic_1.o src/dynamic_1.c
Creating object output/dynamic_2.o from src/dynamic_2.c
gcc -c -o output/dynamic_2.o src/dynamic_2.c
Creating object output/dynamic_3.o from src/dynamic_3.c
gcc -c -o output/dynamic_3.o src/dynamic_3.c
gcc -shared -o output/lib.so output/static_1.o output/static_2.o output/dynamic_1.o output/dynamic_2.o output/dynamic_3.o
rm src/dynamic_1.c src/dynamic_2.c src/dynamic_3.c

However, with parallel building (-j 2), the dynamic_* files are generated twice:

mkdir -p output
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
Creating object output/static_1.o from src/static_1.c
gcc -c -o output/static_1.o src/static_1.c
Creating object output/static_2.o from src/static_2.c
gcc -c -o output/static_2.o src/static_2.c
Creating object output/dynamic_1.o from src/dynamic_1.c
gcc -c -o output/dynamic_1.o src/dynamic_1.c
Creating object output/dynamic_2.o from src/dynamic_2.c
gcc -c -o output/dynamic_2.o src/dynamic_2.c
Creating object output/dynamic_3.o from src/dynamic_3.c
gcc -c -o output/dynamic_3.o src/dynamic_3.c
gcc -shared -o output/lib.so output/static_1.o output/static_2.o output/dynamic_1.o output/dynamic_2.o output/dynamic_3.o
rm src/dynamic_1.c src/dynamic_2.c src/dynamic_3.c

How should I change the makefile to generate the files only one? A clue can be here, but I haven't found a way to use it: multiple targets from one recipe and parallel execution

The project to download is here: http://stuff.pitris.info/make.tar.gz

Thanks!

score 1 · Accepted Answer · answered Nov 07 '14 at 13:23

That linked answer is the answer. Your rule:

$(SRC_DIR)/dynamic_%.c: definition_for_generator.xml
    @echo "---------------------------------------"
    @echo "Generating dynamic Cs"
    @echo "---------------------------------------"
    ./generate.sh $<

Tells make that a single $(SRC_DIR)/dynamic_%.c file is created by each invocation of generate.sh. However running that script actually generates multiple files (but make doesn't know that).

So when you run make normally make operates sequentially and when it sees the first dynamic_*.c file it runs the script to generate that single file (but all the files are generated). When make then goes to determine if the next dynamic_*.c file needs to be generated it sees that the file already exists and skips the rule for generating it.

When you run make -j make spawns parallel processes and those just so happen to each hit a dynamic_*.c file at the same time and so they all run the the script (even though only one of them needs to).

That's why you see it running N times. Change your target order/etc. and you may see fewer generate.sh runs as the parallelism works out differently (you might even manage to race it to only run once) but that's all coincidental.

The right solution, as in that linked answer, is to teach make that all the files are created by running the script by listing multiple patterns for that rule. Or, alternatively, modify generate.sh to only generate a single file at a time (and always run it N times).

The simplest (though somewhat ugly) way to do the multiple patterns thing would likely be to replace

GEN_SRCS = dynamic_1.c dynamic_2.c dynamic_3.c

with

GEN_SRCS = %ynamic_1.c %ynamic_2.c %ynamic_3.c

or something like that and then replace

$(SRC_DIR)/dynamic_%.c: definition_for_generator.xml

with

$(GEN_SRCS): definition_for_generator.xml

score 0 · Answer 2 · answered Nov 07 '14 at 16:15

There is another issue with your code: dependency on $(OUTPUT) directory. It never works as expected.

The below makefile works as expected, please try:

SRC_DIR = src
OUTPUT = output

TARGET = $(OUTPUT)/lib.so

GEN_SRCS = dynamic_1.c dynamic_2.c dynamic_3.c
SRCS = static_1.c static_2.c $(GEN_SRCS)

ifeq ($(filter clean,$(MAKECMDGOALS)),)
-include $(OUTPUT)/.timestamp
endif

OBJS = $(SRCS:%.c=$(OUTPUT)/%.o)

mkdir_@ = $(if $(wildcard $(@D)),,mkdir -p $(@D))

all: $(TARGET)

$(TARGET): $(OBJS)
    gcc -shared -o $@ $^

$(OUTPUT)/%.o: $(SRC_DIR)/$*%.c
    $(mkdir_@)
    @echo "Creating object $@ from $^"
    gcc -c -o $@ $<

$(OUTPUT)/.timestamp: definition_for_generator.xml
    $(mkdir_@)
    @echo "---------------------------------------"
    @echo "Generating dynamic Cs"
    @echo "---------------------------------------"
    ./generate.sh $<
    touch $@

.PHONY: clean
clean:
    rm -rf $(OUTPUT) $(SRC_DIR)/dynamic_1.c $(SRC_DIR)/dynamic_2.c $(SRC_DIR)/dynamic_3.c

The idea is to inject dependency on generated empty makefile - $(OUTPUT)/.timestamp . This is to force GNU Make reread you Makefile. The rule to generate $(OUTPUT)/.timestamp runs your source generator. Thus when GNU Make starts processing your Makefile the second time, generated sources are in place already. Thus you don't need to define any dependencies for them, i.e. they are like your regular source files.

Hint: writing generated sources into source tree is not a good idea preventing use of this source tree by multiple builds at the same time. This is the most obvious issue with this approach.

Also look at mkdir_@ function: it will efficiently create directory for the target if it doesn't exist first trying to use GNU Make builtin to check if the directory exists before spawning mkdir process. Makes a difference when used with Cygwin as process creation in this environment is very time consuming operation compared to the first class UNIX systems.

Output:

$ make -j
mkdir -p output
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
touch output/.timestamp
Creating object output/static_1.o from src/static_1.c
gcc -c -o output/static_1.o src/static_1.c
Creating object output/static_2.o from src/static_2.c
gcc -c -o output/static_2.o src/static_2.c
Creating object output/dynamic_1.o from src/dynamic_1.c
Creating object output/dynamic_2.o from src/dynamic_2.c
gcc -c -o output/dynamic_1.o src/dynamic_1.c
gcc -c -o output/dynamic_2.o src/dynamic_2.c
Creating object output/dynamic_3.o from src/dynamic_3.c
gcc -c -o output/dynamic_3.o src/dynamic_3.c
gcc -shared -o output/lib.so output/static_1.o output/static_2.o output/dynamic_1.o output/dynamic_2.o output/dynamic_3.o

Parallel make generates code multiple times

2 Answers2