Refactoring CTest

Development
1

Recently, I wrote down some ideas how I think CTest should be refactored.

This summarizes the refactoring steps:
https://purplekarrot.net/blog/refactoring-ctest.html

I also wrote some introduction about that topic in a blog post called “Building and Testing with CMake”, but as a new user, I am not allowed to put more than two links here…

This is a draft of the first refactoring step, explained in the blog post above.
https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9867/diffs

I am very open for comments. I am also open for sponsoring, if you consider that work useful.

4 Likes
2

@purpleKarrot thanks for looking at this. The conventions around CTest’s role as a dashboard client changed several times over the years, so the internals are a patchwork of varying styles. It will be nice to get that cleaned up!

1 Like
3

The –build-config argument should be changed to --config like it is used with cmake.

The --config and -C should be common options with same sematic.

P.S: It is also inconsistent in cpack?

see too

2 Likes
4

I like the idea of this refactoring since it seemed such a burden to setup the dashboard mode without clear benefits unless you integrate fully with cdash, but even then the navigability seems wonky. I have a few comments:

  • Regarding ctest --build-and-test, this mode seems more for functional testing CMake implementation, e.g. I use it to test that the packaging works as expected when my project is imported via find_package, FetchContent, etc.
  • How does the cmake --workflow fit into this?
  • Keep in mind that there are multiple audience for the ctest functionality. I primarily split them as: upstream developer, packagers and end-users
    • upstream developer: probably the only audience for the dashboard functionality with coverage, and other such development tools
    • packagers: primarily using plain ctest commands because they need fine-grained control over configure/build/install steps. The refactoring of the dashboard mode should not be towards making that be the primary definition, e.g. having some variable defined only in the CTestScript.cmake that are expected to be used in the normal ctest execution
    • end-users: they primarily care to just test that the build or current package works as intended. A particular design that I encourage is to have a test sub-project that can either be add_subdirectory as part of the current build or be used as a standalone to test pre-installed projects with find_package. One thing to note here is accounting for the different build workflows, and how to allow end-users to use the bundled CTestScript.cmake for their usecase as well