Parallelization of recursive jobs in GNU make

Question

I am looking for an elegant way for the parallelization of jobs in GNU make. Here is a sample of what I did so far. Make processes the directories dir-1, dir-2 and dir-3 in a serial fashion which is logical but not my intention:

SUBDIRS=dir-1 dir-2 dir-3

default: all

all:
  @for dir in $(SUBDIRS); do (cd $$dir; $(MAKE)); done

.PHONY: clean

clean:
  @for dir in $(SUBDIRS); do (cd $$dir; $(MAKE) clean); done

Is there a way to support parallel processing of these directories using the "-j" option without specifying specific targets for each directory?

score 17 · Accepted Answer · edited Jan 07 '14 at 16:40

17

SUBDIRS = a b c

default: all

$(SUBDIRS)::
    $(MAKE) -C $@ $(MAKECMDGOALS)

all clean : $(SUBDIRS)

edited Jan 07 '14 at 16:40

tshepang

12,111
21
91
136

answered Nov 05 '09 at 15:03

mr grumpy

1,513
1
10
17

3

Explanation on double-colons: https://www.gnu.org/software/make/manual/html_node/Double_002dColon.html – csl Oct 30 '14 at 18:54
1

An explanation WHY is double-colon rule needed here would be nice. Having read the description, it isn't. – Victor Sergienko Jan 16 '19 at 01:24
1

So it always executes, even when the subdir exists. – mr grumpy Jan 17 '19 at 22:34

Laurynas Biveinis · Answer 2 · 2020-03-25T06:25:26.760

5

This probably will not answer your question directly, but besides what the other answers suggest, I would recommend to look into non-recursive make techniques. That is truly an elegant way to parallelize build, although, depending on what are the existing Makefiles, can require significant effort. Non-recursive make has advantages not limited to easy parallelization: it sees a complete dependency graph and so it does not need to build too little or too much, meaning faster (sometimes much faster) build times.

Some resources:

edited Mar 25 '20 at 06:25

answered Nov 05 '09 at 17:30

Laurynas Biveinis

10,547
4
53
66

Thx for these ressources. I'll put some effort into getting to know how make works. Actually I am not that familiar with it. – dubbaluga Nov 06 '09 at 12:24
1

Design url is 404'ing – eagle33322 Mar 24 '20 at 16:46
1

http://sites.e-advies.nl/nonrecursive-make.html might be an updated url – eagle33322 Mar 24 '20 at 17:24
Yeah - it looks like it's the right updated URL, I have updated the answer – Laurynas Biveinis Mar 25 '20 at 06:25

score 4 · Answer 3 · answered Apr 30 '12 at 21:07

I found that when using the :: (double-colon rule), I could not find a way to force order if there were any dependencies among app, lib, and doc. After reading much of the GNU Make manual, I came up with the following rules to enforce the dependencies at the same time as having a generic recursive make. Notice that the .PHONY rule is there to force make to enter the directory even though the directory already exists.

SUBDIRS = app lib doc

default: all

app: lib

.PHONY: $(SUBDIRS)

$(SUBDIRS):
    $(MAKE) -C $@ $(MAKECMDGOALS)

all clean install: $(SUBDIRS)

Dirk Eddelbuettel · Answer 4 · 2009-11-05T16:13:33.470

3

Are dir-1, dir-2 and dir-3 interdependent or independent?

I have a similar structure but dependence between the subdirectories so with that I preferred to just use parallel builds within each of the subdirectories. You'd get that via

## default to four parallel runs
MAKEFLAGS += -j 4  

all:
  @for dir in $(SUBDIRS); do (cd $$dir; $(MAKE) ); done

But another trick is to read up on SUBDIRS in the make manual -- you do not need the for loop as make can unroll this for you. Try something like

## default to four parallel runs
MAKEFLAGS += -j 4  

SUBDIRS =    dir-1 dir-2 dir-3

$(SUBDIRS):  #whatever your depends here
             $(MAKE) -C $@

edited Nov 05 '09 at 16:13

answered Nov 05 '09 at 14:52

Dirk Eddelbuettel

360,940
56
644
725

2

It's best not to hard code '-j 4' inside the Makefile. It will be passed to the child make via MAKEFLAGS when called with $(MAKE) if '-j n' is specified by the user on the command line. – jmanning2k Nov 05 '09 at 15:01
Agreed in principle, in practice this was from a real Makefile where I am too lazy to always type 'make -j 4' yet want parallel builds -- and also see my note about the interdependence between sub-directories. – Dirk Eddelbuettel Nov 05 '09 at 15:15
Try this - sets a default but doesn't clobber command line: MAKEFLAGS += -j 4 – jmanning2k Nov 05 '09 at 16:08
Do you know how make handles these $(MAKEFLAGS)? I think the number of simultaneous jobs is the number of processed directories multiplied by 4 in this case, or something like that. Thus, I would not forward the "jobs"-argument to child-makes. What do you think? – dubbaluga Nov 06 '09 at 12:27
That is not my impression. I mod'ed an existing project with five SUBDIRS to use 'MAKEFLAGS += -j 4' and I got up to four compilations going. – Dirk Eddelbuettel Nov 06 '09 at 14:38

Parallelization of recursive jobs in GNU make

4 Answers4

Linked