Testing client cmake usage

Code
#1

We want our clients to be able to find_package our package, and we supply a number of sample applications in our repository. We would like to be able to test that these build and run correctly from the perspective of a client. Ie. we don’t want to add the projects via add_subdirectory because that might, eg. hide a target we failed to export to the installation.

Is there a good way within CMake to automate this, preferably such that CTest can manage “building and running apps” as a set of tests unto itself? I’m aware of ExternalProject, which seems promising, but it seems to need to download the sources from another URL.

#2

You probably want to make use of ctest’s build-and-test mode. Something like this:

add_test(NAME ClientBuild
  COMMAND ${CMAKE_CTEST_COMMAND} --build-and-test
          ClientBuildProj
          ${CMAKE_CURRENT_BINARY_DIR}/ClientBuildProj
          --build-generator ${CMAKE_GENERATOR}
          --build-options -DCMAKE_PREFIX_PATH=XXX
)

where XXX is the path to the area you have arranged to put your installed artefacts (you might have your build tree already mimicking that install layout, or you could use another test case to set it up and define a test fixture to ensure it always runs before this test case).

Inside the ClientBuildProj directory, you would put a CMakeLists.txt which is representative of what a client project would need to do to consume your project.

#3

This is great! Thanks so much, Craig!

#4

One quick follow-up: is it possible to use CPack / my existing install targets to do this? ie. is it safe to run cmake --build . --target package on my own project from a test case?

It looks like cmake --install is only supported on 3.15+ and the newest version I can use (thanks to CI) is 3.14

#5

Test cases should not invoke the main build. Tests could run in parallel and if the build actually needs to build something (i.e. not just install/package things), then you could have multiple builds running simultaneously and that is both unsupported and likely to end in a big mess.

See the end of the documentation page for the install() command (look for the Generated Installation Script section). It shows a way to invoke the install using the cmake -P script mode instead. It was only documented recently, but it works for quite a long way back through past CMake versions.

1 Like
#6

That worked perfectly. Thanks again!

#7

Followup question, many months later:

--build-and-test mode apparently considers the entire subdirectory’s set of tests to be a single ‘test’. This is not at all unreasonable, of course, but our use of it has turned up some suboptimal behavior that I haven’t found a workaround for.

In our case, we have a subproject being invoked this way that runs a number of benchmarks as part of its “test”, which can take a loooooong time if you don’t have your system configured correctly (leading some users to assume there is a hang). In fact, progress is being made, it’s just that (1) all the tests and benchmarks in the subdir are running serially and must all complete, and (2) there doesn’t appear to be any way to get build-and-test mode to emit output from the sub-test to the enclosing test (at least, nothing I’ve found) until it completes.

In my ideal world, there would be some way for --build-and-test to have the sub-tests participate in the enclosing test command (ie, with the same granularity in terms of parallelism, timeouts, etc); I’m going to guess that this is unlikely, but I’ll ask anyway just in case there’s a hidden option I’ve missed.

Failing that, is there at least some way to get --build-and-test to flush test output periodically, so that there’s indication that something is happening?

#8

(1) all the tests and benchmarks in the subdir are running serially and must all complete

In your --test-command, you specify the ctest command line. You should be able to pass -j or --parallel to let its tests run non-serially. You may have to hard-code how many parallel jobs to use, in which case you probably also want to set the outer test’s PROCESSORS test property so you don’t over-subscribe the machine’s cpus. You might also be able to do something that uses the CTEST_PARALLEL_LEVEL environment variable instead of using the -j or --parallel option, see what works for you.

(2) there doesn’t appear to be any way to get build-and-test mode to emit output from the sub-test to the enclosing test (at least, nothing I’ve found) until it completes.

This is the case with all test in general I think. You can give -V or -VV options to the outer or inner ctest command to get it to print more output as it is generated.