Hello all,
I am trying my hand at cdash to get more feedback from our nightly build pipeline.
To do this I am using a ctest script and am I using the “all-at-once” method. i followed this explanation: https://www.kitware.com/cdash-all-at-once-subproject-builds/
Project structure:
root
-> CMakeLists.txt
/src
-> CMakeLists.txt
/tests
-> CMakeLists.txt
/acceptancetests
-> CMakeLists.txt
In the tests/CMakeLists.txt I added: set(CMAKE_DIRECTORY_LABELS “unit_tests”) before calling the add_subdirectories and add_executable.
Similarly, in the acceptancetests/CMakeLists.txt I added: set(CMAKE_DIRECTORY_LABELS “acceptance_tests”) before calling the add_subdirectories and add_executable.
In src/CMakeLists.txt I called set(CMAKE_DIRECTORY_LABELS “library”), then add all my add_subdirectories. Then I call set(CMAKE_DIRECTORY_LABELS “executable”) before calling the add_subdirectory that creates the main executable target.
My cmake test script looks like:
# Note: this cannot be combined with the CMakePresets => we need to redefine ALL
# Note: auth tokens expire every 6 months. see CTestConfig.cmake file
include(${HOME}/CTestConfig.cmake)
site_name(CTEST_SITE)
set(CTEST_BUILD_NAME ${CMAKE_HOST_SYSTEM_NAME})
set(CTEST_SOURCE_DIRECTORY ${HOME})
set(CTEST_BINARY_DIRECTORY ${HOME}/out/build/ctest)
set(CTEST_CMAKE_GENERATOR Ninja)
set(CTEST_CONFIGURATION_TYPE RelWithDebInfo)
set(XMGR_TEST_CONFIG_FLAGS
"-DCMAKE_CXX_FLAGS_INIT=-fsanitize=address -fno-omit-frame-pointer"
"-DCMAKE_EXE_LINKER_FLAGS_INIT=-fsanitize=address -fno-omit-frame-pointer"
)
set(CTEST_LABELS_FOR_SUBPROJECTS executable library unit_tests acceptance_tests)
# PERFORM TEST
# =============
ctest_empty_binary_directory(${CTEST_BINARY_DIRECTORY})
ctest_start("Experimental")
ctest_configure(OPTIONS "${configureOpts}")
ctest_submit(PARTS Start Configure HTTPHEADER ${CDASH_SUBMIT_HEADER})
ctest_build()
ctest_submit(PARTS Build HTTPHEADER ${CDASH_SUBMIT_HEADER})
ctest_test()
ctest_submit(PARTS Test HTTPHEADER ${CDASH_SUBMIT_HEADER})
if(XMGR_WITH_COVERAGE AND CTEST_COVERAGE_COMMAND)
ctest_coverage()
ctest_submit(PARTS Coverage HTTPHEADER ${CDASH_SUBMIT_HEADER})
endif()
if(XMGR_WITH_MEMCHECK AND CTEST_MEMORYCHECK_COMMAND)
ctest_memcheck()
ctest_submit(PARTS MemCheck HTTPHEADER ${CDASH_SUBMIT_HEADER})
endif()
ctest_submit(PARTS Submit HTTPHEADER ${CDASH_SUBMIT_HEADER})
if(NOT CMAKE_VERSION VERSION_LESS "3.14")
ctest_submit(PARTS Done HTTPHEADER ${CDASH_SUBMIT_HEADER})
endif()
I am publishing to the my.cdash.org website, but to a private project.
Roughly speaking, the current project has 36 unit tests and 17 acceptance tests.
But in my CDash dashboard I see:
So all 53 tests get assigned to the unit testing label.
My guess is this has something to do with the test discovery in ctest. I am using Catch2, so I don’t explicitly call add_test, but I use the catch_discover_tests().
I call catch_discover_tests() twice, once for acceptance tests and once for unit tests, each time with a different target as argument.
If I run those targets separatly, then each runs only their own tests, but if I run ctest it always mixes the unit and acceptance tests (even if I specify only the unit testing target). My guess is that ctest does not have the logic to separate the tests?
My question is: can I get the reporting separated while still doing the all-in-one strategy? Or do I need to split up and configure + build twice to get the information separated?
Actually, the same goes for the other steps. As you notice the ‘warn’ tab in the build has all the warnings only on the unit_tests label, while in truth the warnings are all over the place (yes, something on my todo list)
all with CMake 4.0.3