MSVC10 /MP builds not multicore across folders in a project

Question

I'm hoping someone points out something wrong or a workaround for what we are experiencing.

When compiling a project with /MP it appears that only files in the same folder are compiled concurrently. I used process explorer to swipe the command line and confirm the behavior.

Project filters seem to have all no impact on the what gets compiled concurrently.

Project structure on disk:

Folder\
  project.vcxproj  
  source\  
    foo.cpp  
    foo1.cpp  
  other_folder\  
    bar.cpp
    bar1.cpp
    bar3.cpp

Initial process tree:

MSBuild.exe
  cl.exe  ( passed: source\foo.cpp source\foo1.cpp )
    cl.exe  ( passed: source\foo.cpp )
    cl.exe  ( passed: source\foo1.cpp )

After the 2 child instances of cl.exe complete the parent closes and the following process tree appears:

MSBuild.exe
  cl.exe  ( passed: other_folder\bar.cpp other_folder\bar1.cpp other_folder\bar2.cpp )
    cl.exe  ( passed: other_folder\bar.cpp )
    cl.exe  ( passed: other_folder\bar1.cpp )
    cl.exe  ( passed: other_folder\bar2.cpp )

Our source is nicely organized in many levels of nested folders that match the layout of the headers on disk - I would hate to have to give that up to take advantage of /MP.

Answer 1

The use of %(RelativeDir) in "Object File Name" ( in the vcxproj XML, /Fo on the CL.exe command line ) project causes msbuild to batch the cpp files to cl.exe on a per directory basis. This can have a significant impact on the benefits gained from using /MP.

Note that if your project uses %(RelativeDir) for object files its likely that the configuration is trying to avoid colliding .obj files that from cpp files with the same name in different folders.

The /Fo command line parameter is typically a folder that the compiler dumps the obj files into - there is only ONE passed, so all of the cpp files for a given directory can only be passed to CL.exe at a time.

That was a pain - but I'm glad there is a reason and a solution. Hope it helps.

---

Update

A team mate found that anytime an MSBuild parameter is sent to CL.exe it seems to break or severely limit /MP. This is most likely because for /MP to work well the top level CL.exe needs to have a bundle of cpp files.

Our solution was to not use any msbuild params ( I think its the %params% ) for 'Object File Name'. This required that we rename some cpp files so they did not collide.

Hope this has changed in VS2012 or VS2013.

Answer 2

I've addressed this issue here with a new approach for avoiding .obj clobbering conflicts

https://stackoverflow.com/a/26935613/4253427

With the above in mind, I can elaborate on "anytime an MSBuild parameter is sent to CL.exe it seems to break or severely limit /MP". /MP works one CL.exe invocation at a time. When you "send an msbuild parameter" what you actually do is create multiple CL.exe invocations with commandlines tailored to each source file. These can't be batched. The above linked solution attempts to work around that by tailoring a commandline to a minimal set of output directories.

This should solve your problem by preventing clobbers while maintaining a high degree of batching, so long as your project doesn't contain 100 files all named x.cpp in different directories... usually in my experience there are only a few colliding .obj in a much larger project.

Answer 3

According it MSDN, the files should be compiled when ever there is a thread to handle them, it at the same time makes no guarantee's about the order in which files are to be compiled:

The source files might not be compiled in the same order in which they appear on the command line. Although the compiler creates a set of processes that contain copies of the compiler, the operating system schedules when each process executes. Consequently, you cannot guarantee that the source files will be compiled in a particular order.

A source file is compiled when a process is available to compile it. If there are more files than processes, the first set of files is compiled by the available processes. The remaining files are processed when a process finishes handling a previous file and is available to work on one of the remaining files.

It also states that the number of created process will be bound by the number of threads and files in the command line:

That value is the lesser of the number of source files that you specify on the command line

Combining these to things, we can see that the compiler handles the compilation incrementally(file wise) so that it can dispatch work to children correctly, it dose this folder by folder.

You might be able to get around this if you generated a custom make file, where you should be able to get more than one folder being processed at the same time (or try using the MSBUILD.exe tool).

Answer 4

Can confirm the issue, and the cause wasn't easy to find, so let's get some more keywords in for other users stumbling on it.

I've been noticing that rebuilds on my latest MSVC project take way too long. In particular, the CPU cores are barely utilized, the number on cl.exe processes in Task Manager varies wildly and the Output window shows the source files being compiled in what seem like batches. Anywhere from one to 16 are being compiled at a time, a small pause, then another set of files, and so on. In comparison, on my older projects the CPU is pretty much fully used and the Output window shows a continuous stream of source files being compiled.

Now, the big difference in my new project was making a better use of namespaces with a matching directory structure, which meant ending up with some classes having identical names and causing conflicts due to the different .obj files heading for the same directory, which lead to changing the Object File Name in C/C++ -> Output Files.

Updates to the Output window also match the directory structure. If there's a namespace/directory with one source file inside it, then VS shows just that one file being compiled at a time. The next directory has 10 source files and VS shows all 10 being compiled simultaneously.

There aren't many solutions. Either avoid classes with the same name and don't change the Object File Name, or use the the workaround posted by zeromus, it functions great. My rebuild time went from 03:15 to 01:20, which is quite a difference and matches the CPU utilization going from ~35% to a flatline 100% during most the compilation.

VS 2015, 2017 and 2019 all behave in this way, so not much hope for a change.