test does not run with configuration

Code
1

I added a test which does not run:

add_test( NAME debugOnly
    COMMAND Cap24::testapp
    CONFIGURATIONS Debug
)

the test should be executed only for the ‘Debug’ configuration.
I am using the gnu makefiles generator
I am generating my build with Debug configuration:

cmake -D CMAKE_BUILD_TYPE=Debug ..

I am building the test with debug configuration:

cmake --build . --target test --config Debug

in CMake cache we can confirm I am using Debug configuration:

//Choose the type of build, options are: None Debug Release RelWithDebInfo
// MinSizeRel ...
CMAKE_BUILD_TYPE:STRING=Debug

I expect the test to run.
Why does the test not run?

2

I don’t think the test build target has those kinds of controls. I imagine you probably need to use ctest -C Debug to get these things working.

3

I can confirm it does work using ctest instead. It is counter intuitive that the configuration is not being passed on to ctest per default, especially when explicitly passing the configuration on as argument.

This only applies to single configuration generators. For multi configuration generators the currently selected build type is passed on to ctest with the RUN_TESTS target.

Is there a reason for this behaviour?
Do you agree this would be a reasonable improvement request?

4

cmake --target test is just “run the test target in the build system”. There’ s no mechanism for passing extra arguments to that command based on the other command line arguments to cmake --build.

That’s because the multi-config generators have a per-config test target to run. You can’t change the configuration of a single-config generator without rerunning CMake anyways. That is, building in Release only offers the Release tests; a Debug test run needs a Debug build.

5

that makes it more complicated. So to make a test run @ debug only I need to make sure its a multi config build. Otherwise, I must make sure the build type is debug before actually adding the test. Finally, the same differentiation must be made when using ctest.

Based on this I think the following is required:

if( NOT isMultiConfig )
    if( isDebug )
        add_test( NAME debugOnly
            COMMAND Cap24::testapp
            CONFIGURATIONS Debug
        )
    endif()
else()
    add_test( NAME debugOnly
        COMMAND Cap24::testapp
        CONFIGURATIONS Debug
    )
endif()

it would, however, be much more convenient to simply add the test to CMake:

add_test( NAME debugOnly
    COMMAND Cap24::testapp
    CONFIGURATIONS Debug
)

which is then run with cmake --target test if and only if current active ${CMAKE_BUILD_TYPE} is "Debug".

If a Debug build needs a debug build, what’s the point of ctest not knowing there’s a Debug build?

6

The documentation of add_test says:

CONFIGURATIONS
Restrict execution of the test only to the named configurations.

This means you don’t have to check whether CMAKE_BUILD_TYPE is Debug or whether Debug is part of CMAKE_CONFIGURATION_TYPES. That test will only be executed if the configuration is Debug.

7

I think what I’d do in this case is to always make the test, but trigger RETURN_SKIP_CODE in non-Debug builds. I don’t know that I’ve ever seen add_test(CONFIGURATIONS) used before (and based on this behavior, it seems like not many others have either).

You can do this with either COMMAND Cap24::testapp $<CONFIG> or doing a -Dconfig=$<CONFIG> compile definition.

8

using RETURN_SKIP_CODE is indeed more convenient.

again, the current behaviour doesn’t seem very convenient. There doesn’t seem to be any draw backs passing the configuration type on to ctest for single configuration generators, just like it is the case for multi configuration generators.

Given the current behaviour it makes sense it isn’t used by many, but it still seems to be a useful feature. I can imagine a number of things I’d test on a debug build which I would not on a release.

An alternative to test configurations could be test grouping.