--parallel does not really enable parallel compiles with msbuild?

Usage
,
#1

Somebody warned me against using (or trusting) the --parallel option on the command line recently saying that it always sets cl_mpcount to 1 when using msbuild. I tried and sure enough this is true. After reading what cl_mpcount is, it seems like --parallel isn’t doing what it claims to do for visual studio generators. Is there something I’m missing? Shouldn’t the --parallel option pass /p:CL_MPcount=N where N is the number of virtual processors detected?

#2

--parallel option currently maps to the /m flag as that was originally the only parallel flag offered by Visual Studio. So you get multiple projects being built in parallel but not multiple compilation units from the same project.

#3

Do you know of any plans to update this? As it is today it covers IMO the less-useful parallelism case. If not I can submit a feature request.

#4

The CMake code says:

      // Having msbuild.exe and cl.exe using multiple jobs is discouraged
      makeCommand.Add("/p:CL_MPCount=1");

(https://gitlab.kitware.com/cmake/cmake/-/blob/v3.17.1/Source/cmGlobalVisualStudio10Generator.cxx#L1082-1083).

So it seems to be done on purpose, I guess to avoid over-using the CPUs.

#5

I tihnk the settings should be reversed, at least based on the C++ projects I’ve seen. Perhaps MSBuild is geared with two “knobs” like this because C# projects are different and have more project parallelism (only guessing)? In my experience C++ projects benefit more from running multiple instances of cl within one project than allowing multiple projects to be processed together.

I would propose maximizing CL_MPcount and setting /m:1. This IMO most closely reflects how --parallel works in other environments.

#6

This is achievable for many years, long before --parallel was added to cmake --build, by adding /MP to the compile options and not passing any flag to cmake --build/MSBuild.

#7

Yes, that’s perfectly valid. Of course you’re right. I’m saying for projects that are maybe not your own (e.g. building a dependency), it’s arguably easier to not have to modify that top-level CMakeLists.txt with something like this:

add_compile_options($<$<CXX_COMPILER_ID:MSVC>:/MP>)

Ultimately, I feel it would be nice to have better consistency across platforms in how --parallel works.

1 Like
#8

I totally agree!

I recently tried using --parallel in a project of mine and had to give up because xcodebuild requires a number of jobs, but:

  • if I give a low number (2 or 4), I won’t get maximum parallelization, and
  • if I give a high number (6 or 8), I might exhaust all memory on CI nodes.

So I sticked to -j4 when using make, -maxcpucount when using MSBuild, and nothing when using xcodebuild :man_shrugging:

1 Like
#9

Good point. I think it’s great just to have a common option (–parallel / -j) now across various compilers/build tools. If I have to put a number in there versus using the auto-determined number that’s not a huge loss to me.

#10

I’ll try making this a formal request - if it gets thru great, if not I can always fall back on the harder-to-remember:

cmake --build . --  /p:CL_MPcount=8
#11

For reference, this has been raised as CMake Issue 20564.

1 Like
#12

If we are going to update this I think the new mtt flag ( https://devblogs.microsoft.com/cppblog/improved-parallelism-in-msbuild/ ) that does auto balancing between multi-TU and multi-project would be the best approach

2 Likes