Automatically setting jobs (-j) flag for a multicore machine?

Question

I have a Makefile on a machine that has a ton of cores in it, but I always seem to forget to write -jX when compiling my project and it takes way longer than it should.

Is there some way I can set the -j flag through an environment variable or some other persistent config file so that make will automatically execute multiple jobs in parallel on this machine?

Answer 1

I'm assuming you're using Linux. This is from my ~/.bashrc

# parallel make
export NUMCPUS=`grep -c '^processor' /proc/cpuinfo`
alias pmake='time nice make -j$NUMCPUS --load-average=$NUMCPUS'

sample usage

samm@host src> echo $NUMCPUS
8
samm@host src> pmake

becomes time nice make -j8 --load-average=8.

To answer your specific question about putting this into a Makefile, I don't find it practical to sprinkle this logic all over my Makefiles. Putting this into a top level Makefile also isn't a great solution since I often build from sub-directories and wish to build them in parallel as well. However, if you have a fairly flat source hierarchy, it may work for you.

Answer 2

It appears that the MAKEFLAGS environment variable can pass flags that are part of every make run (at least for GNU make). I haven't had much luck with this myself, but it might be possible to use -l rather than -j to automatically run as many jobs as are appropriate for the number of cores you have available.

Answer 3

As Jeremiah Willcock said, use MAKEFLAGS, but here is how to do it:

export MAKEFLAGS="-j $(grep -c ^processor /proc/cpuinfo)"

or you could just set a fixed value like this:

export MAKEFLAGS="-j 8"

If you really want to boost performance you should use ccache by adding something like the following to your Makefile:

CCACHE_EXISTS := $(shell ccache -V)
ifdef CCACHE_EXISTS
    CC := ccache $(CC)
    CXX := ccache $(CXX)
endif

Answer 4

I usually do this as follows in my bash scripts:

make -j$(nproc)

Answer 5

Aliases are not expanded inside scripts. It's better to create a separate make script and place it into one of the $PATH directories:

#!/bin/sh

if [ -f /proc/cpuinfo ]; then
    CPUS=`grep processor /proc/cpuinfo | wc -l`
else
    CPUS=1
fi
/usr/bin/make -j`expr $CPUS + 1` "$@"

Answer 6

You can add a line to your Makefile similar to the following:

NUMJOBS=${NUMJOBS:-" -j4 "}

Then add a ${NUMJOBS} line in your rules, or add it into another Makefile var (like MAKEFLAGS). This will use the NUMJOBS envvar, if it exists; if it doesn't, automatically use -j4. You can tune or rename it to your taste.

(N.B.: Personally, I'd prefer the default to be -j1 or "", especially if it's going to be distributed to others, because although I have multiple cores also, I compile on many different platforms, and often forget to dis-able the -jX setting.)

Answer 7

Until sometime in 2016, you could put this in your makefile: (GNU make tested)

MAKEFLAGS += "-j$(NUM_CORES) -l$(NUM_CORES)

(where NUM_PPROCS is calculated or set according to one of many of the other answers here) And, bam! you have multi-process building going on.

Given that this has stopped working, the best thing that I could come up with is this, where the makefile calls itself, but with -jX and -lX.

ifeq ($(PARALELL_WRAPPER_ABXCOEOEKCOEBMQJKHTOEUB),done)

all: ...
   ...

other_target: ...
    ...

else
# add parallelism equal to number of cores every time.
# "random" strings are to ensure uniqueness
NUM_CORES ?= $(shell grep -c "vendor_id" /proc/cpuinfo)
MAKEFLAGS +=" -j$(NUM_CORES) -l$(NUM_CORES) "

# for the default target case
parallel_wrapper_default_target_anthsqjkshbeohcbmeuthnoethoaeou:
    $(MAKE) PARALELL_WRAPPER_ABXCOEOEKCOEBMQJKHTOEUB=done

# catches everything else
% :
    $(MAKE) $@ PARALELL_WRAPPER_ABXCOEOEKCOEBMQJKHTOEUB=done

endif

Answer 8

On Ubuntu 16.4 using all CPU cores:

export MAKEFLAGS='-j$(nproc)'

or

export MAKEFLAGS='-j 2'

Answer 9

At the beginning of a Makefile:

MAKEFLAGS+="j"

It won't take numeric arguments for any version of GNU Make before 4.2. After 4.2 you can do:

MAKEFLAGS+="j2"

For versions earlier than 4.2 if you happen to have some jobs running out of memory, you could make them run only one at a time with flock from util-linux-ng. For example, a convert utility from ImageMagick will use all resources it can get when used to optimize images according to Google's recommendations, so there's no point to have it run in parallel.

%.min.jpg: %.jpg
    @flock --wait 600 Makefile convert $< -sampling-factor 4:2:0 -strip $@

It is important to set a long wait time because make will still run most of these commands in parallel. Therefore, wait time must be as such as the deepest queue execution time. If you have eight cores, and eight large images take a minute to optimize, you must set the wait time to at least a minute.

With hundreds of cores and huge images, six hundred seconds set above might not be enough.

Answer 10

I would just put

alias make='make -j'

in ~/.profile or ~/.bashrc.

According to the manual:

If there is nothing looking like an integer after the ‘-j’ option, there is no limit on the number of job slots.