Integrating CDash submission into a normal CI

Usage
, ,
1

Hi, I’m having trouble integrating CDash into my project’s CI and haven’t found any good answers, so I’m asking here.

I manage a pretty standard CI that configures, builds, and tests a CMake project for a range of configurations. This project handles the entire test setup as part of the CMakeLists.txt, there is no CTestConfig in the root folder.

How do I:

  1. specify the site name and build name using the command line or environment variables?
  2. execute configuration, build, and tests as individual steps in the CI. I want to see quickly at which stage the CI fails. This rules out running everything as a single script.
  3. use CDash without a CTestConfig in the root?

Thanks a lot!

1 Like
2

In practice, I do place a CTestConfig.cmake file in the repo root.

My CI workflows typically run something like ctest -D Nightly, but you could break it up and run each test phase (update, configure, build, test) as a separate step in your CI, ex:

  • ctest -D NightlyStart
  • ctest -D NightlyUpdate
  • ctest -D NightlyConfigure

etc. See the cli docs.

3

XRootD uses a custom script to submit from GitHub Actions to CDash.
You can find the script here:

and the GitHub Actions workflow using it is here:

It should be quite easy to adapt it to your project. I hope you find it useful.

Best regards,
—Guilherme

4

Do you generate this as part of the CI?

@amadio Thanks. This is a very good reference for a full script that ticks most of our boxes. The message grouping was on our mind, too. It helps to make the output a bit easier to read, but it doesn’t run individual steps.

5

No, why would you? It’s not a mortal sin for there to be a CTestConfig.cmake file sitting in the repo root of any normal git clone. It has project-specific information, so it belongs in the project, IMHO.

6

[quote=“Ben Vining, post:5, topic:15587, username:benthevining”]
No, why would you?… It has project-specific information, so it belongs in the project, IMHO.
[/quote]

Some setting are run-specific though for example the name of the site/CI runner or the name of the build including configuration details etc.

Form my understanding, they also need to be part of the CTestConfig and it cannot depend on variables from the CMakeLists nor CMakeCache, so I need to patch or generate that file. And I don’t really want to patch files in the source directory based on some configuration in the binary directory.

7

My script runs the steps in sequence with the split sections for messages, as you pointed out, and always does a ctest_submit at the end, without running subsequent steps if a step fails. The reason I decided to run the steps together is that the submit step needs the build name and other things which I detect in the beginning, so if I did the steps separately, I’d have to rerun parts of the script multiple times. You can, however, use CTEST_ARG or optional arguments to customize that script to run only certain steps. I have an optional step that installs the project, which is enabled with -DINSTALL=1, for example. Similarly for enabling clang-tidy, sanitizers, valgrind, etc. There’s also a .ci/config.cmake with default options, and per-OS detection for customizations. You could use that for your various configurations.

For CTestConfig.cmake, we have one in the project’s top directory. If you need customizations per build, you could use environment variables in the set commands within your CTestConfig.cmake, or use a CTestConfig.cmake.in to generate one in the build directory or have it done by the CI as you said.

Have a look here for an easier to follow description of our CI:

Cheers,

8

No, the CTestConfig file contains only the following content:

set (CTEST_PROJECT_NAME ben-bot)
set (CTEST_NIGHTLY_START_TIME 01:00:00 UTC)
set (CTEST_SUBMIT_URL https://my.cdash.org/submit.php?project=ben-bot)
set (CTEST_DROP_SITE_CDASH TRUE)

All other settings should be specified elsewhere

9

The CTestConfig.cmake isn’t picked up in the build directory (CMake 3.31.6).

OK. I think I finally figured out the chain of information:

  1. The ctest calls rely on the DartConfiguration.tcl
  2. The DartConfiguration.tcl is generated when include(CTest) is called.
  3. The CTest module reads the CTestConfig.cmake in the root of the source directory and what the documentation refers to as CTest module variable.

So, setting up CDash support without a standalone ctest script is possible via

  • Populating the CTest module variables in CMakeLists.txt before calling include(CTest). This can be done by
    • adding a CTestConfig.cmake to the root of your source directory,
    • programmatically in the CMakeLists.txt using set(),
    • by passing these via the CLI using -D, or
    • by prepopulating the cache using -C.
  • Skip include(CTest) and directly generate the DartConfiguration.tcl following the format specification.