# test does not run with configuration

**URL:** <https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018>\
**Category:** Code\
**Tags:** gen:makefiles\
**Created:** [April 14, 2020, 12:49pm UTC](https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018 "2020-04-14T12:49:47Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![hex](https://discourse.cmake.org/user_avatar/discourse.cmake.org/hex/32/363_2.png) [@hex](https://discourse.cmake.org/u/hex)\
**Post date:** [April 14, 2020, 12:49pm UTC](https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018/1 "2020-04-14T12:49:47Z")

</div>

I added a test which does not run:

```cmake
add_test( NAME debugOnly
    COMMAND Cap24::testapp
    CONFIGURATIONS Debug
)

```

the test should be executed only for the ‘Debug’ configuration.  
I am using the gnu makefiles generator  
I am generating my build with Debug configuration:

```bash
cmake -D CMAKE_BUILD_TYPE=Debug ..

```

I am building the test with debug configuration:

```bash
cmake --build . --target test --config Debug

```

in CMake cache we can confirm I am using Debug configuration:

```auto
//Choose the type of build, options are: None Debug Release RelWithDebInfo
// MinSizeRel ...
CMAKE_BUILD_TYPE:STRING=Debug

```

I expect the test to run.  
Why does the test not run?

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [April 14, 2020, 1:22pm UTC](https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018/2 "2020-04-14T13:22:27Z")

</div>

I don’t think the `test` build target has those kinds of controls. I imagine you probably need to use `ctest -C Debug` to get these things working.

---

<div class="post-metadata">

**Author:** ![hex](https://discourse.cmake.org/user_avatar/discourse.cmake.org/hex/32/363_2.png) [@hex](https://discourse.cmake.org/u/hex)\
**Post date:** [April 14, 2020, 3:59pm UTC](https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018/3 "2020-04-14T15:59:12Z")

</div>

I can confirm it does work using `ctest` instead. It is counter intuitive that the configuration is not being passed on to ctest per default, especially when explicitly passing the configuration on as argument.

This only applies to single configuration generators. For multi configuration generators the currently selected build type is passed on to `ctest` with the `RUN_TESTS` target.

Is there a reason for this behaviour?  
Do you agree this would be a reasonable improvement request?

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [April 14, 2020, 6:00pm UTC](https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018/4 "2020-04-14T18:00:38Z")

</div>

> [@hex](#):
>
> Is there a reason for this behaviour?

`cmake --target test` is just “run the `test` target in the build system”. There’ s no mechanism for passing extra arguments to that command based on the other command line arguments to `cmake --build`.

> [@hex](#):
>
> For multi configuration generators the currently selected build type is passed on to `ctest` with the `RUN_TESTS` target.

That’s because the multi-config generators have a per-config `test` target to run. You can’t change the configuration of a single-config generator without rerunning CMake anyways. That is, building in Release only offers the Release tests; a Debug test run needs a Debug build.

---

<div class="post-metadata">

**Author:** ![hex](https://discourse.cmake.org/user_avatar/discourse.cmake.org/hex/32/363_2.png) [@hex](https://discourse.cmake.org/u/hex)\
**Post date:** [April 14, 2020, 6:53pm UTC](https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018/5 "2020-04-14T18:53:14Z")

</div>

> [@ben.boeckel](#):
>
> a Debug test run needs a Debug build

that makes it more complicated. So to make a test run @ debug only I need to make sure its a multi config build. Otherwise, I must make sure the build type is debug before actually adding the test. Finally, the same differentiation must be made when using `ctest`.

Based on this I think the following is required:

```cmake
if( NOT isMultiConfig )
    if( isDebug )
        add_test( NAME debugOnly
            COMMAND Cap24::testapp
            CONFIGURATIONS Debug
        )
    endif()
else()
    add_test( NAME debugOnly
        COMMAND Cap24::testapp
        CONFIGURATIONS Debug
    )
endif()

```

it would, however, be much more convenient to simply add the test to CMake:

```cmake
add_test( NAME debugOnly
    COMMAND Cap24::testapp
    CONFIGURATIONS Debug
)

```

which is then run with cmake --target test if and only if current active `${CMAKE_BUILD_TYPE}` is `"Debug"`.

If a Debug build needs a debug build, what’s the point of `ctest` not knowing there’s a Debug build?

---

<div class="post-metadata">

**Author:** ![McMartin](https://discourse.cmake.org/user_avatar/discourse.cmake.org/mcmartin/32/150_2.png) [@McMartin](https://discourse.cmake.org/u/McMartin)\
**Post date:** [April 14, 2020, 7:28pm UTC](https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018/6 "2020-04-14T19:28:47Z")

</div>

The documentation of `add_test` says:

> `CONFIGURATIONS`  
> Restrict execution of the test only to the named configurations.

This means you don’t have to check whether `CMAKE_BUILD_TYPE` is `Debug` or whether `Debug` is part of `CMAKE_CONFIGURATION_TYPES`. That test will only be executed if the configuration is Debug.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [April 14, 2020, 7:28pm UTC](https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018/7 "2020-04-14T19:28:54Z")

</div>

I think what I’d do in this case is to always make the test, but trigger `RETURN_SKIP_CODE` in non-Debug builds. I don’t know that I’ve ever seen `add_test(CONFIGURATIONS)` used before (and based on this behavior, it seems like not many others have either).

You can do this with either `COMMAND Cap24::testapp $<CONFIG>` or doing a `-Dconfig=$<CONFIG>` compile definition.

---

<div class="post-metadata">

**Author:** ![hex](https://discourse.cmake.org/user_avatar/discourse.cmake.org/hex/32/363_2.png) [@hex](https://discourse.cmake.org/u/hex)\
**Post date:** [April 14, 2020, 7:57pm UTC](https://discourse.cmake.org/t/test-does-not-run-with-configuration/1018/8 "2020-04-14T19:57:41Z")

</div>

using `RETURN_SKIP_CODE` is indeed more convenient.

> [@ben.boeckel](#):
>
> and based on this behavior, it seems like not many others have either

again, the current behaviour doesn’t seem very convenient. There doesn’t seem to be any draw backs passing the configuration type on to `ctest` for single configuration generators, just like it is the case for multi configuration generators.

Given the current behaviour it makes sense it isn’t used by many, but it still seems to be a useful feature. I can imagine a number of things I’d test on a debug build which I would not on a release.

An alternative to test configurations could be test grouping.