# add\_test() has different behaviour for VS and Ninja generator

**URL:** https://discourse.cmake.org/t/add-test-has-different-behaviour-for-vs-and-ninja-generator/1046
**Category:** Usage
**Created:** [April 17, 2020, 1:25pm UTC](https://discourse.cmake.org/t/add-test-has-different-behaviour-for-vs-and-ninja-generator/1046 "2020-04-17T13:25:18Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![KUGA2](https://discourse.cmake.org/user_avatar/discourse.cmake.org/kuga2/32/4968_2.png) [@KUGA2](https://discourse.cmake.org/u/KUGA2)
#### Post date: [April 17, 2020, 1:25pm UTC](https://discourse.cmake.org/t/add-test-has-different-behaviour-for-vs-and-ninja-generator/1046/1 "2020-04-17T13:25:18Z")

</div>

I have this CMakeList:

```
add_test(
   NAME with_command_and_configuration
   COMMAND ${CMAKE_COMMAND} --version
   CONFIGURATIONS one another
)

```

Now I configure it. It results in different ctest code:

```
MINGW64 /c/Temp/ctest_configurations
$ cd default && rm -rf * && cmake ..; cd ..
-- Building for: Visual Studio 14 2015
-- Selecting Windows SDK version to target Windows 10.0.18363.
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Temp/ctest_configurations/default

MINGW64 /c/Temp/ctest_configurations
$ cat default/CTestTestfile.cmake | grep with_command_and_configuration

MINGW64 /c/Temp/ctest_configurations
$ cd ninja && rm -rf * && cmake .. -G Ninja; cd ..
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Temp/ctest_configurations/ninja

MINGW64 /c/Temp/ctest_configurations
$ cat ninja/CTestTestfile.cmake | grep with_command_and_configuration
  add_test(with_command_and_configuration "C:/ProgramData/chocolatey/lib/cmake.portable/tools/cmake-3.17.1-win64-x64/bin/cmake.exe" "--version")
  set_tests_properties(with_command_and_configuration PROPERTIES _BACKTRACE_TRIPLES "C:/Temp/ctest_configurations/CMakeLists.txt;9;add_test;C:/Temp/ctest_configurations/CMakeLists.txt;0;")

```

Is this a bug? Or is this somehow correct and related to the Multi-Configuration nature of the VS Generator?

PS: full project and CTestTestfile.cmake in attachment[ctest\_configurations.zip](https://discourse.cmake.org/uploads/short-url/eKzafTPlofvFsOMeixp9HHba6TU.zip) (1.9 KB)

PPS: I found this in 3.10, but the logs are of:

```
MINGW64 /c/Temp/ctest_configurations
$ cmake --version
cmake version 3.17.1
```

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [April 17, 2020, 2:38pm UTC](https://discourse.cmake.org/t/add-test-has-different-behaviour-for-vs-and-ninja-generator/1046/2 "2020-04-17T14:38:16Z")

</div>

> [@KUGA2](#):
>
> CONFIGURATIONS one another

The VS generators know the available configurations at generate time, so I suspect these are getting dropped because `one` and `another` are not in `CMAKE_CONFIGURATION_TYPES`. Since it knows there’s no configuration available for the test, it gets dropped.

For single-config generators, there’s nothing actually tying `CMAKE_BUILD_TYPE` to the `ctest -C` flag (more specifically, `ctest` doesn’t look up the test configuration from `CMakeCache.txt`), so it leaves the filtering until test time (which you can see with the `if` guard on the test you’re looking at).

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [April 17, 2020, 2:38pm UTC](https://discourse.cmake.org/t/add-test-has-different-behaviour-for-vs-and-ninja-generator/1046/3 "2020-04-17T14:38:45Z")

</div>

See also this thread: [CTest build-and-test with multi-config generators](https://discourse.cmake.org/t/ctest-build-and-test-with-multi-config-generators/1008)

---

<div class="post-metadata">

### Author: ![KUGA2](https://discourse.cmake.org/user_avatar/discourse.cmake.org/kuga2/32/4968_2.png) [@KUGA2](https://discourse.cmake.org/u/KUGA2)
#### Post date: [April 17, 2020, 3:21pm UTC](https://discourse.cmake.org/t/add-test-has-different-behaviour-for-vs-and-ninja-generator/1046/4 "2020-04-17T15:21:45Z")

</div>

Thank you. So my hunch was right there.

However this is still inconsistent. A single-config generator then should filter out all test but the ones with the (default) `CMAKE_BUILD_TYPE` Configuration!?!

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [April 17, 2020, 4:07pm UTC](https://discourse.cmake.org/t/add-test-has-different-behaviour-for-vs-and-ninja-generator/1046/5 "2020-04-17T16:07:36Z")

</div>

Eh. Not necessarily. The reason is that multi-config generator paths are generally independent of the build configuration, so the rules can be written out for any unknown configuration just fine. This is not the case for multi-config generators, so they end up getting skipped.