By default CMake’s test target doesn’t depend on all which leaves many users scratching their heads when ‘make test’ fails because the project isn’t built yet.
We can use set(CMAKE_SKIP_TEST_ALL_DEPENDENCY FALSE) to fix this, but we need to paste this set in every top-level CMakeLists.txt that’s attempting to follow best practices.
Is there appetite to switch the default behavior to have test depend on all via a policy change?
test not depending on all has been the convention since CMake started. It is considered a convenience for running ctest, and running ctest directly would not build all.
CMAKE_SKIP_TEST_ALL_DEPENDENCY was added in CMake 3.29 as an option.
As maintainer, I’m not particularly interested in changing the default that has been established for over 20 years.
To pick up on another aspect of this, I’d recommend moving away from using the test build target at all and instead execute ctest directly. Most mainstream IDEs are doing so these days, and it offers a far richer and more flexible set of capabilities, especially for developers working directly from a command line. The days of make test stem from an era that lacked much of today’s testing features (I’m not just talking about CMake projects in that statement).
CMAKE_SKIP_INSTALL_ALL_DEPENDENCY exist too, but I miss the information when it was added.
P.S: I use both options to create more complex cmake workflow presets.
And ctest has the option –build-and-test…
That option has existed since 2005.
Thanks for that context.
That’s reasonable for the rationale you cited.
I’m concluding that it is not a best practice to include a set(CMAKE_SKIP_TEST_ALL_DEPENDENCY FALSE) line in every top-level CMakeLists.txt because:
ctest directly is preferred over usage of the test build target anyway, andctest directly which is contradicted by set(CMAKE_SKIP_TEST_ALL_DEPENDENCY FALSE) semantics.But how will you handle it test depends an install, and install depends an all?
There’s a way to make a ctest test that depends on the install target having been executed? Interesting, that’s the first I’ve heard of that.
… but the error messages are note really helpfully:
bash-5.2$ ctest --preset dev
Test project /Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev
Start 1: find-package-test
Start 2: header-only-test
Could not find executable /Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
Looked in the following places:
/Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
/Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
/Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/Debug/header-only-test
/Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/Debug/header-only-test
Debug//Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
Debug//Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/Debug/header-only-test
Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/Debug/header-only-test
Debug/Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
Debug/Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
Unable to find executable: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/header-only-test
1/3 Test #2: header-only-test .................***Not Run 0.00 sec
Start 3: module_header-only-test
2/3 Test #3: module_header-only-test .......... Passed 0.03 sec
3/3 Test #1: find-package-test ................***Failed 1.00 sec
Internal cmake changing into directory: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/tests/find-package-test
Error: cmake execution failed
The CXX compiler identification is Clang 19.1.4
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Check for working CXX compiler: /usr/local/opt/llvm/bin/clang++ - skipped
Detecting CXX compile features
Detecting CXX compile features - done
CMake Error at CMakeLists.txt:9 (find_package):
By not providing "Findfmt.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "fmt", but
CMake did not find one.
Could not find a package configuration file provided by "fmt" (requested
version 11.0.2.1) with any of the following names:
fmtConfig.cmake
fmt-config.cmake
Add the installation prefix of "fmt" to CMAKE_PREFIX_PATH or set "fmt_DIR"
to a directory containing one of the above files. If "fmt" provides a
separate development package or SDK, be sure it has been installed.
Configuring incomplete, errors occurred!
33% tests passed, 2 tests failed out of 3
Total Test time (real) = 1.00 sec
The following tests FAILED:
1 - find-package-test (Failed)
2 - header-only-test (Not Run)
Errors while running CTest
bash-5.2$
does not help if you call ctest without previous cmake --build ...
But the cmake workflow preset helps:
bash-5.2$ cmake --workflow dev
Executing workflow step 1 of 4: configure preset "dev"
Preset CMake variables:
CMAKE_BUILD_TYPE="Debug"
CMAKE_CXX_EXTENSIONS="ON"
CMAKE_CXX_FLAGS="-fstack-protector-strong -Wall -Wextra -Wpedantic -Wno-conversion -Wno-sign-conversion -Wcast-qual -Wformat=2 -Wundef -Werror=float-equal -Wshadow -Wcast-align -Wunused -Wnull-dereference -Wdouble-promotion -Wno-implicit-fallthrough -Wextra-semi -Woverloaded-virtual -Wnon-virtual-dtor -Wno-old-style-cast"
CMAKE_CXX_STANDARD="23"
CMAKE_CXX_STANDARD_REQUIRED="ON"
CMAKE_EXPORT_COMPILE_COMMANDS="ON"
CMAKE_INSTALL_PREFIX:PATH="/Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir"
CMAKE_PREFIX_PATH:STRING="/Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir"
FMT_DEVELOPER_MODE="ON"
-- use ccache
-- CPPdefinitions=FMT_MODULE
-- CPM: Adding package fmt@11.0.2 (11.0.2 at /Users/clausklein/.cache/CPM/fmt/cccb77ae9609d2768ed80dd42cec54f77b1f1455)
-- CPM: Adding package googletest@1.15.2 (v1.15.2 at /Users/clausklein/.cache/CPM/googletest/63d366a111d035db343a038884a22e44986dcf4d)
-- Module tests to be built: header-only-test
-- Configuring done (0.1s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev
Executing workflow step 2 of 4: build preset "dev"
[9/13] Building CXX object CMakeFiles/fmt.dir/module/fmt.cppm.o
In file included from /Users/clausklein/Workspace/cpp/cxx20/fmt-module/module/fmt.cppm:129:
/Users/clausklein/.cache/CPM/fmt/cccb77ae9609d2768ed80dd42cec54f77b1f1455/include/fmt/os.h:33:14: warning: '#include <filename>' attaches the declarations to the named module 'fmt', which is not usually intended; consider moving that directive before the module declaration [-Winclude-angled-in-module-purview]
33 | # include <fcntl.h> // for O_RDONLY
| ^
1 warning generated.
[12/13] Install the project...
-- Install configuration: "Debug"
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/args.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/base.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/chrono.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/color.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/compile.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/core.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/format-inl.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/format.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/os.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/ostream.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/printf.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/ranges.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/std.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/include/fmt/xchar.h
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/libfmtD.a
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/module/fmt.cppm
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/module/format.cc
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/module/os.cc
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/fmtConfig.cmake
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/fmtConfigVersion.cmake
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/fmtTargets.cmake
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/fmtTargets-debug.cmake
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/./cxx-modules-fmtTargets.cmake
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/./cxx-modules-fmtTargets-Debug.cmake
-- Installing: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/stagedir/lib/cmake/fmt/./target-fmt-Debug.cmake
Executing workflow step 3 of 4: test preset "dev"
Test project /Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev
Start 1: find-package-test
Start 2: header-only-test
Start 3: module_header-only-test
1/3 Test #3: module_header-only-test .......... Passed 0.00 sec
2/3 Test #2: header-only-test ................. Passed 0.52 sec
3/3 Test #1: find-package-test ................ Passed 4.00 sec
100% tests passed, 0 tests failed out of 3
Total Test time (real) = 4.00 sec
Executing workflow step 4 of 4: package preset "dev"
CPack: Create package using TGZ
CPack: Install projects
CPack: - Install project: fmt []
CPack: Create package
CPack: - package: /Users/clausklein/Workspace/cpp/cxx20/fmt-module/build/dev/fmt-11.0.2-Darwin.tar.gz generated.
bash-5.2$