Feature Request: Give me a way to specify the MSVC Toolset when using "-G Ninja"

Usage
, ,
1

TL;DR: I wish CMake had an optional arg that meant “run vcvarsall x64|x86|etc before doing anything else”.

More backstory:

  • I have a buildbot worker script running in a git-for-windows bash shell (basically, msys2)
  • The buildmaster can send it compile/test tasks that require pulling source from somewhere, rebuilding it, and testing it.
  • A given buildbot worker may be given tasks that are built for x86 or x64 (so we can’t just start a shell that has the right vcvarsall call in place, we may need to change the active MSVC compiler for each subsequent task)
  • To date, we’ve been using something like -G Visual Studio 16 2019 -T host=x64 -A Win32 (or -A x64) when configuring CMake, and everything is fine… except of course that this uses MSBuild, which is very very slow.
  • We’d like to switch to using -G Ninja, because it is crazy-faster, but AFAICT there isn’t an equivalent to the -A argument for that Generator; to make this work, we have to explicitly call vcvarsall.bat ourselves.
  • Calling vcvarsall.bat ourselves is doable but ugly, because we’re running in git-for-windows-bash (rather than cmd.exe); we can do something like vsvarsall.bat x64 && set, slurp the env vars dumped via set, massage the file paths into a form that won’t make bash choke (due to backslashes, etc), set them in the environment… and do this before every call to CMake (for either config or for build). This can be made to work, but man oh man is it ugly and fragile.

So…
(1) Is there a better way to do this already?
(2) If not, can CMake somehow add the smarts (that it must already have) to allow the Ninja generator to get the same tool-selection treatment that the VS generator has?

1 Like
2

Use a regular cmd prompt and use call path\to\vcvarsall.bat x64 -vcvars_ver=toolsetversion. This does the equivalent of source on the script.

The Ninja (and other non-IDE generators) expect to be run in an environment with the toolchain already available. It’s just that the MSVC toolchain expects to work from a complicated environment and not just a simple CC= environment setting. There’s probably some deeper reasoning for it not being supported for all generators, but I suspect it has to do with how much control CMake actually has over it (likely none, so it disclaims all of it).

3

Use a regular cmd prompt

As mentioned before, we’re using a bash prompt (via git-for-windows) as parts of our setup use bashisms that would require bifurcation of scripts to make things work via cmd.exe, so this isn’t a great option for us.

1 Like
4

You did not mention how your build command looks like but adding build tool specific options may speed up the build process a lot even for MSBuild.
By default MSBuild does not try to build in parallel. You may add add_compile_definitions(/MP) for MSVC to your CMakeLists.txt to enable that, but it can also be done using MSBuild command line arguments. For building our solutions with MSBuild in parallel we are using the following command

cmake --build builddir --target ALL_BUILD --config Release -- /nologo /verbosity:minimal /maxcpucount:2 /property:MultiProcessorCompilation=true /l:FileLogger,Microsoft.Build.Engine;logfile=logfilename.log

Hope that helps.

5

You can probably write a cmd or PowerShell script and launch that from bash if you want to keep it as your entry point.

6

We’re using parallel builds with MSBuild.exe (typically 2 or 4 depending on the build machine); when building with Ninja using the same values (via -j) the Ninja builds are still considerably faster (especially partial rebuilds).

7

Sure – but CMake must already have some logic built in for this (since it can apparently set up the environment correctly for the VS Generator, based on the -A flag). Would be nice to be able to re-use that for Ninja too.

8

Our setup is simply telling MSBuild what we want. It knows how to make the environment work.

9

@steven-johnson

The following module may be useful to discover the MSVC toolchain:

We use it to find the appropriate vcvars script and build external projects that do not use CMake.

A simple CMake script could be created that would use this module and allow you to do this:

cmake -P /path/to/ConfigureWithVisualStudioToolchain.cmake  ^
  -DGENERATOR=Ninja ^
  -DPROJECT_SOURCE_DIR=/path/to/src ^
  -DPROJECT_BINARY_DIR=/path/to/bld

To select a specific visual studio toolchain, you would pass options like -DVcvars_MSVC_VERSION=... -DVcvars_MSVC_ARCH= ....

The script ConfigureWithVisualStudioToolchain.cmake would be something like this:

Note: I did not test the following code

# Sanity check
# TODO: Check that expected parameters are set ... 

set(download_dir ${PROJECT_BINARY_DIR}/CMakeFiles/FindVcvars)
file(MAKE_DIRECTORY ${download_dir})

# Download FindVcvars.cmake
set(dest_file "${download_dir}/FindVcvars.cmake")
set(expected_hash "cacfcb6e243e4a0fa628c5873d0cb444c08079b7729af24093a59712b01b6ff6")
set(url "https://raw.githubusercontent.com/scikit-build/cmake-FindVcvars/v1.3/FindVcvars.cmake")
if(NOT EXISTS ${dest_file})
  file(DOWNLOAD ${url} ${dest_file} EXPECTED_HASH SHA256=${expected_hash})
else()
  file(SHA256 ${dest_file} current_hash)
  if(NOT ${current_hash} STREQUAL ${expected_hash})
    file(DOWNLOAD ${url} ${dest_file} EXPECTED_HASH SHA256=${expected_hash})
  endif()
endif()

list(INSERT CMAKE_MODULE_PATH 0 ${download_dir})
find_package(Vcvars REQUIRED)

# TODO: Support passing argument like toolset, ... 

execute_process(COMMAND
  ${Vcvars_LAUNCHER} ${CMAKE_COMMAND} -G ${GENERATOR} -S ${PROJECT_SOURCE_DIR} -B ${PROJECT_BINARY_DIR}
  )