95

Is it somehow possible to be able to have a parallel build no matter which build tool is used?

Under Unix we can add make -jN where N are the number of threads, and under Windows I added to the CXX_FLAG "/MP" which is then used in Visual Studio to parallel build...(?) How can I make my version such that CMAKE_MAKE_PROGRAM is not always extended when I run CMake?

What is a general solution?

I came up with this:

# Add some multithreaded build support
MARK_AS_ADVANCED(MULTITHREADED_BUILD)
set(MULTITHREADED_BUILD 12 CACHE STRING "How many threads are used to build the project")
if(MULTITHREADED_BUILD)
    if(${CMAKE_GENERATOR} MATCHES "Unix Makefiles")
            message(STATUS ${CMAKE_BUILD_TOOL})
            set(CMAKE_MAKE_PROGRAM "${CMAKE_MAKE_PROGRAM} -j${MULTITHREADED_BUILD}")
            message(STATUS "Added arguments to CMAKE_BUILD_TOOL: ${CMAKE_MAKE_PROGRAM}")
    elseif(MSVC)
      set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /MP")
      message(STATUS "Added parallel build arguments to CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
    endif()
endif()
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Gabriel
  • 8,990
  • 6
  • 57
  • 101
  • 1
    Not sure it can be done, don't think its a good idea anyway. CMake describes how the project is build and how make does its thing is a different level from CMake. – Lap May 22 '12 at 16:08
  • Exetnding your approach just add `-DMULTITHREADED_BUILD=12` to the command line of cmake. and remove the `SET(MULTITHREADED_BUILD ...)` – bikram990 May 14 '13 at 11:36
  • In my case with Eclipse CDT results in error: "/usr/bin/make -j8" all Cannot run program "/usr/bin/make -j8": Unknown reason Error: Program "/usr/bin/make -j8" not found in PATH – Hendy Irawan Apr 06 '15 at 03:28
  • My CMake / Eclipse CDT error was due to CMAKE_MAKE_PROGRAM bug: http://www.cmake.org/Bug/view.php?id=15497 – Hendy Irawan Apr 06 '15 at 04:06
  • 2
    As this post is a little bit old, users of CMake >= 3.12.0 should consider the answer of @usr1234567 – dkg Oct 11 '18 at 12:43

6 Answers6

96

With CMake 3.12 this is possible. From the release notes:

The cmake(1) Build a Project (cmake --build) gained --parallel [<jobs>] and -j [<jobs>] options to specify a parallel build level. They map to corresponding options of the native build tool.

As mentioned by dkg, you can also set the environment variable CMAKE_BUILD_PARALLEL_LEVEL.

Links to CMake's documentation:

Mike T
  • 41,085
  • 18
  • 152
  • 203
usr1234567
  • 21,601
  • 16
  • 108
  • 128
  • 8
    You can also set the env var `CMAKE_BUILD_PARALLEL_LEVEL` – dkg Oct 11 '18 at 12:40
  • 6
    What does "They map to corresponding options of the native build tool" exactly mean? For example it does not seem to automatically set the `/MP` flag for MSVC when using CMake 3.16.4 and Ninja 1.10.0. So it seems for a true/full parallel build there is still no cross-platform solution available. – Florian Wolters Feb 10 '20 at 16:47
24

If you have CMake v2.8.8 or higher, you may use Ninja as an alternative of GNU make:

mkdir build
cd    build
cmake -G Ninja ..
ninja              # Parallel build (no need -j12)

or

mkdir build
cd    build
cmake -G Ninja ..
cmake --build .    # Parallel build using Ninja

As you can see, no need to use CMAKE_MAKE_PROGRAM, the build is run in parallel by default, optimizing the number of jobs depending on available CPU cores.

Ninja is based on a low-level JSON configuration to speed up the startup phase. Therefore its JSON configuration is not easy to write by hand, and I always generate it using a high-level tool/IDE:

As a C++ build often requires lots of memory, your computer must provide as much memory as the number of CPU cores.

As pointed out by Ruslan, CMake 3.12 (2018) has a new option cmake --build -j <N> to limit build to <N> cores (jobs) thus limiting the memory consumption (see also the documentation). If you use an older CMake version, you can still use cmake --build -- -j <N>. The option -- tells to CMake to pass the rest directly to the underlying builder tool, here it is Ninja.

oHo
  • 51,447
  • 27
  • 165
  • 200
  • 2
    Regarding the last sentence: you can limit number of cores used with `-j N` option, thus the amount of memory used. – Ruslan Oct 03 '19 at 14:20
  • 1
    Thank you Ruslan Three years after my answer, CMake has finally the option `-j N` I was waiting for. Have fun – oHo Oct 05 '19 at 23:35
  • 1
    Before CMake 3.12 that you could just pass this option to Ninja — either running it directly as in your first snippet, or appending `-- -jN` after `cmake --build` (this `--` feature of `--build` exists even in CMake 2.8.0). – Ruslan Oct 06 '19 at 07:00
  • Thank you @Ruslan . I was aware about the `--` option that I used with `make`. But I never used it with `ninja`. Maybe in 2015, `ninja` had no option `-j `... or I was not using it correctly... I have updated the answer to fit your experience. Thanks – oHo Oct 14 '19 at 12:51
  • using the new cmake -j option if passing `-j 0` it will use all the cores automatically, like an autodetect the max cores availables and use them all? – Raffaello Nov 29 '21 at 12:23
14

As already mention above one can use --parallel [<jobs>] (or -j [<jobs>]) option of CMake to build solution in parallel.

But I would like to note that in Windows a command

cmake.exe --build --parallel <jobs>

starts a root MSBuild process with the following arguments:

msbuild.exe /m:<jobs> /p:CL_MPCount=1 <another-useful-arguments...>

This means the root MSBuild process will start multiple child MSBuild processes due to /m:<jobs> (or /maxcpucount:<jobs>). But each of this child MSBuild runs at most one compiler process due to /p:CL_MPCount=1 argument (for more details find this link). In other words each MSBuild can compile at most one source file at the same time.

To overcome this limitation one may call CMake in the following way:

cmake.exe --build --parallel <n_msbuild> -- /p:CL_MPcount=<n_cl>

This approach sets /MP option thereby allows multiple compiler processes under each MSBuild instance.

And for the record. To configure builds ran form Visual Studio GUI one need to go to Settings:

  • Option "Projects and Solutions → Build And Run → maximum number of parallel projects build" affects on maxcpucount.
  • Option "Projects and Solutions → VC++ Project Settings → Build → Maximum Concurrent C++ Compilations" represents the default boundary to /MP.
Nikita
  • 191
  • 1
  • 4
10

Now it is very simple to parallel build with cmake. You can add "-j jobs_number" when using "cmake --build". For example:

cmake --build . -j 24

More details can be found in CMAKE manual: https://cmake.org/cmake/help/latest/manual/cmake.1.html#build-tool-mode

--parallel [], -j [] The maximum number of concurrent processes to use when building. If is omitted the native build tool’s default number is used.

The CMAKE_BUILD_PARALLEL_LEVEL environment variable, if set, specifies a default parallel level when this option is not given.

Some native build tools always build in parallel. The use of value of 1 can be used to limit to a single job.

cunzhang
  • 197
  • 2
  • 4
2

You can't do this cross-platform. The -jN option is a parameter to make, and not part of the generated Makefile. However, you could have CMake generate a Bash script that runs make for your project using -jN (where the script looks up the number of cores you have).

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
bossbarber
  • 870
  • 6
  • 10
  • 5
    For Visual Studio: `msbuild your_project.sln /maxcpucount:N` – Jean Davy May 02 '15 at 20:41
  • 4
    @JeanDavy: You can even pass this parameter using the CMake build tool by calling `cmake --build . -- /maxcpucount:8` as CMake passes all parameters following the "empty" `--` to the build tool. – Roland Sarrazin Jul 26 '17 at 16:09
  • 1
    @RolandSarrazin yep, my current script is `cmake --build . --target install --config release -- /m` and it seems to be working on all cores. Thanks to [this discussion](https://cmake.org/pipermail/cmake/2011-November/047784.html). I also suggest to not specify exact number, in this case MSBuild will consider that build must be performed on all available cores, but not more than exist. – Alex Zhukovskiy Feb 03 '18 at 13:08
  • 1
    No longer true. Beginning with 3.12, CMake offers according flags. See my answer. – usr1234567 Jul 13 '18 at 05:19
1

I have settled down to writing a parallelmake.sh script for Unix Makefiles-based generators. This is done here: https://github.com/gabyx/ApproxMVBB

And the relevant parts in the the CMake file:

https://github.com/gabyx/ApproxMVBB/blob/master/CMakeLists.txt#L89

# Add some multithreaded build support =====================================================================================================
MARK_AS_ADVANCED(MULTITHREADED_BUILD)
SET(MULTITHREADED_BUILD ON CACHE BOOL "Parallel build with as many threads as possible!")
if(MULTITHREADED_BUILD)
    if(${CMAKE_GENERATOR} MATCHES "Unix Makefiles")
            file(COPY ${ApproxMVBB_ROOT_DIR}/cmake/parallelmake.sh DESTINATION ${PROJECT_BINARY_DIR}
                FILE_PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE
                NO_SOURCE_PERMISSIONS
            )
            SET(CMAKE_MAKE_PROGRAM "${PROJECT_BINARY_DIR}/parallelmake.sh")
            MESSAGE(STATUS "Set make program to ${PROJECT_BINARY_DIR}/parallelmake.sh")
    elseif(MSVC)
      SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}" "/MP")
      MESSAGE(STATUS "Added parallel build arguments to CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
    endif()
endif()
# ========================================================================================================================================
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Gabriel
  • 8,990
  • 6
  • 57
  • 101