CTest: build dependency as test fixture

Development
1

The following code defines a test executable and two tests:

add_executable(foo_test ...)
add_test(NAME foo.test.1 COMMAND foo_test -1)
add_test(NAME foo.test.2 COMMAND foo_test -2)

With CMAKE_SKIP_TEST_ALL_DEPENDENCY set to OFF, running make test will build the test executable before running the tests, unless EXCLUDE_FROM_ALL is set as a target or directory property. Running ctest directly will not build the test executable at all.

I’d like to discuss an alternative approach that would build the test executable before running the tests that rely on it, whether the tests are launched through ctest or make test and whether EXCLUDE_FROM_ALL is set or not:

If the test command is an executable target, then (based on some variable or property that is yet to be determined) a test fixture is implicitly defined and added as a requirement of the test.
That means, the code

set(CMAKE_COME_UP_WITH_REASONABLE_NAME ON)

add_executable(foo_test ...)
add_test(NAME foo.test.1 COMMAND foo_test -1)
add_test(NAME foo.test.2 COMMAND foo_test -2)

would behave like

add_executable(foo_test ...)
add_test(NAME foo.test.1 COMMAND foo_test -1)
add_test(NAME foo.test.2 COMMAND foo_test -2)

add_test(NAME cmake_build.foo_test
  COMMAND "${CMAKE_COMMAND}"
    --build "${CMAKE_BINARY_DIR}"
    --config $<CONFIGURATION>
    --target foo_test
)

set_tests_properties(cmake_build.foo_test PROPERTIES
  FIXTURES_SETUP cmake_build.foo_test
)
set_tests_properties(foo.test.1 PROPERTIES
  FIXTURES_REQUIRED cmake_build.foo_test
)
set_tests_properties(foo.test.2 PROPERTIES
  FIXTURES_REQUIRED cmake_build.foo_test
)
2

If you try to run a build as part of (or a setup step for) a test, you can no longer run any tests in parallel. You would potentially end up with multiple builds running at the same time for the same build directory.

I don’t think we should be supporting doing a build in the main project’s build directory as part of running tests.

If your proposal is to work out all required targets if the user builds the test target, that’s a different scenario. Personally I think we should be trying to direct users to run ctest directly rather than building the test target though, and I think trying to build required targets as part of the test target will more often than not be not much different to building the default all target. I get it that there will be cases where this is not true, but for the effort involved in implementing it, the value seems questionable (to me, I’m happy to be wrong though if someone else wants to expend that time and effort).

3

Yes, that is convincing.

I think trying to build required targets as part of the test target will more often than not be not much different to building the default all target.

Consider the following usecase:

block()
  set(BUILD_SHARED_LIBS OFF)
  add_subdirectory(bar EXCLUDE_FROM_ALL)
endblock()

add_library(foo)
target_link_libraries(foo PRIVATE bar::bar)

We want to build a third party library as STATIC, irrespective of the setting of BUILD_SHARED_LIBS in the main project. We don’t want to build other parts of the third party project and we especially want to exclude third party components from being installed, hence the EXCLUDE_FROM_ALL.

But now we have a situation where the all target does not build all required targets for testing.

The EXCLUDE_FROM_ALL directory property has this exception:

If a target built by the parent project depends on a target in the subdirectory, the dependee target will be included in the parent project build system to satisfy the dependency.

Do we need another exception for executable targets that are used as the COMMAND in add_test to satisfy the testing dependency? That command already has three special cases when the command is an executable target. Something like:

If <command> specifies an executable target created by add_executable():

  • Add: the exectuable target is included in the ALL target.