add_test() has different behaviour for VS and Ninja generator

Usage
1

I have this CMakeList:

add_test(
   NAME with_command_and_configuration
   COMMAND ${CMAKE_COMMAND} --version
   CONFIGURATIONS one another
)

Now I configure it. It results in different ctest code:

MINGW64 /c/Temp/ctest_configurations
$ cd default && rm -rf * && cmake ..; cd ..
-- Building for: Visual Studio 14 2015
-- Selecting Windows SDK version  to target Windows 10.0.18363.
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Temp/ctest_configurations/default

MINGW64 /c/Temp/ctest_configurations
$ cat default/CTestTestfile.cmake | grep with_command_and_configuration

MINGW64 /c/Temp/ctest_configurations
$ cd ninja && rm -rf * && cmake .. -G Ninja; cd ..
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Temp/ctest_configurations/ninja

MINGW64 /c/Temp/ctest_configurations
$ cat ninja/CTestTestfile.cmake | grep with_command_and_configuration
  add_test(with_command_and_configuration "C:/ProgramData/chocolatey/lib/cmake.portable/tools/cmake-3.17.1-win64-x64/bin/cmake.exe" "--version")
  set_tests_properties(with_command_and_configuration PROPERTIES  _BACKTRACE_TRIPLES "C:/Temp/ctest_configurations/CMakeLists.txt;9;add_test;C:/Temp/ctest_configurations/CMakeLists.txt;0;")

Is this a bug? Or is this somehow correct and related to the Multi-Configuration nature of the VS Generator?

PS: full project and CTestTestfile.cmake in attachmentctest_configurations.zip (1.9 KB)

PPS: I found this in 3.10, but the logs are of:

MINGW64 /c/Temp/ctest_configurations
$ cmake --version
cmake version 3.17.1
2

The VS generators know the available configurations at generate time, so I suspect these are getting dropped because one and another are not in CMAKE_CONFIGURATION_TYPES. Since it knows there’s no configuration available for the test, it gets dropped.

For single-config generators, there’s nothing actually tying CMAKE_BUILD_TYPE to the ctest -C flag (more specifically, ctest doesn’t look up the test configuration from CMakeCache.txt), so it leaves the filtering until test time (which you can see with the if guard on the test you’re looking at).

3

See also this thread: CTest build-and-test with multi-config generators

4

Thank you. So my hunch was right there.

However this is still inconsistent. A single-config generator then should filter out all test but the ones with the (default) CMAKE_BUILD_TYPE Configuration!?!

5

Eh. Not necessarily. The reason is that multi-config generator paths are generally independent of the build configuration, so the rules can be written out for any unknown configuration just fine. This is not the case for multi-config generators, so they end up getting skipped.