# Testing client cmake usage
**URL:** https://discourse.cmake.org/t/testing-client-cmake-usage/643
**Category:** Code
**Created:** [February 17, 2020, 12:41am UTC](https://discourse.cmake.org/t/testing-client-cmake-usage/643 "2020-02-17T00:41:25Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@alex](https://discourse.cmake.org/u/alex)
#### Post date: [February 17, 2020, 12:41am UTC](https://discourse.cmake.org/t/testing-client-cmake-usage/643/1 "2020-02-17T00:41:25Z")
We want our clients to be able to `find_package` our package, and we supply a number of sample applications in our repository. We would like to be able to test that these build and run correctly _from the perspective of a client_. Ie. we don’t want to add the projects via `add_subdirectory` because that might, eg. hide a target we failed to export to the installation.
Is there a good way within CMake to automate this, preferably such that CTest can manage “building and running apps” as a set of tests unto itself? I’m aware of ExternalProject, which seems promising, but it seems to need to _download_ the sources from another URL.
---
### Author:  [@craig.scott](https://discourse.cmake.org/u/craig.scott)
#### Post date: [February 17, 2020, 3:59am UTC](https://discourse.cmake.org/t/testing-client-cmake-usage/643/2 "2020-02-17T03:59:01Z")
You probably want to make use of ctest’s [build-and-test mode](https://cmake.org/cmake/help/latest/manual/ctest.1.html#build-and-test-mode). Something like this:
```cmake
add_test(NAME ClientBuild
COMMAND ${CMAKE_CTEST_COMMAND} --build-and-test
ClientBuildProj
${CMAKE_CURRENT_BINARY_DIR}/ClientBuildProj
--build-generator ${CMAKE_GENERATOR}
--build-options -DCMAKE_PREFIX_PATH=XXX
)
```
where `XXX` is the path to the area you have arranged to put your installed artefacts (you might have your build tree already mimicking that install layout, or you could use another test case to set it up and define a [test fixture](https://cmake.org/cmake/help/latest/prop_test/FIXTURES_REQUIRED.html) to ensure it always runs before this test case).
Inside the `ClientBuildProj` directory, you would put a `CMakeLists.txt` which is representative of what a client project would need to do to consume your project.
---
### Author:  [@alex](https://discourse.cmake.org/u/alex)
#### Post date: [February 17, 2020, 5:14am UTC](https://discourse.cmake.org/t/testing-client-cmake-usage/643/3 "2020-02-17T05:14:19Z")
This is great! Thanks so much, Craig!
---
### Author:  [@alex](https://discourse.cmake.org/u/alex)
#### Post date: [February 19, 2020, 9:06am UTC](https://discourse.cmake.org/t/testing-client-cmake-usage/643/4 "2020-02-19T09:06:33Z")
One quick follow-up: is it possible to use CPack / my existing install targets to do this? ie. is it safe to run cmake --build . --target package on my own project from a test case?
It looks like cmake --install is only supported on 3.15+ and the newest version I can use (thanks to CI) is 3.14
---
### Author:  [@craig.scott](https://discourse.cmake.org/u/craig.scott)
#### Post date: [February 19, 2020, 10:59am UTC](https://discourse.cmake.org/t/testing-client-cmake-usage/643/5 "2020-02-19T10:59:17Z")
> [@alex](#):
>
> is it safe to run cmake --build . --target package on my own project from a test case?
Test cases should not invoke the main build. Tests could run in parallel and if the build actually needs to build something (i.e. not just install/package things), then you could have multiple builds running simultaneously and that is both unsupported and likely to end in a big mess.
> [@alex](#):
>
> It looks like cmake --install is only supported on 3.15+ and the newest version I can use (thanks to CI) is 3.14
See the end of the documentation page for the `install()` command (look for the _Generated Installation Script_ section). It shows a way to invoke the install using the `cmake -P` script mode instead. It was only documented recently, but it works for quite a long way back through past CMake versions.
---
### Author:  [@alex](https://discourse.cmake.org/u/alex)
#### Post date: [February 19, 2020, 11:25am UTC](https://discourse.cmake.org/t/testing-client-cmake-usage/643/6 "2020-02-19T11:25:14Z")
> [@craig.scott](#):
>
> See the end of the documentation page for the `install()` command (look for the _Generated Installation Script_ section). It shows a way to invoke the install using the `cmake -P` script mode instead. It was only documented recently, but it works for quite a long way back through past CMake versions.
That worked perfectly. Thanks again!
---
### Author:  [@steven-johnson](https://discourse.cmake.org/u/steven-johnson)
#### Post date: [November 26, 2020, 12:43am UTC](https://discourse.cmake.org/t/testing-client-cmake-usage/643/7 "2020-11-26T00:43:43Z")
Followup question, many months later:
`--build-and-test` mode apparently considers the entire subdirectory’s set of tests to be a single ‘test’. This is not at all unreasonable, of course, but our use of it has turned up some suboptimal behavior that I haven’t found a workaround for.
In our case, we have a subproject being invoked this way that runs a number of benchmarks as part of its “test”, which can take a loooooong time if you don’t have your system configured correctly (leading some users to assume there is a hang). In fact, progress is being made, it’s just that (1) all the tests and benchmarks in the subdir are running serially and must all complete, and (2) there doesn’t appear to be any way to get build-and-test mode to emit output from the sub-test to the enclosing test (at least, nothing I’ve found) until it completes.
In my ideal world, there would be some way for `--build-and-test` to have the sub-tests participate in the enclosing test command (ie, with the same granularity in terms of parallelism, timeouts, etc); I’m going to guess that this is unlikely, but I’ll ask anyway just in case there’s a hidden option I’ve missed.
Failing that, is there at least some way to get --build-and-test to flush test output periodically, so that there’s indication that something is happening?
---
### Author:  [@craig.scott](https://discourse.cmake.org/u/craig.scott)
#### Post date: [November 28, 2020, 1:48am UTC](https://discourse.cmake.org/t/testing-client-cmake-usage/643/8 "2020-11-28T01:48:22Z")
> (1) all the tests and benchmarks in the subdir are running serially and must all complete
In your `--test-command`, you specify the `ctest` command line. You should be able to pass `-j` or `--parallel` to let its tests run non-serially. You may have to hard-code how many parallel jobs to use, in which case you probably also want to set the outer test’s `PROCESSORS` test property so you don’t over-subscribe the machine’s cpus. You might also be able to do something that uses the `CTEST_PARALLEL_LEVEL` environment variable instead of using the `-j` or `--parallel` option, see what works for you.
> (2) there doesn’t appear to be any way to get build-and-test mode to emit output from the sub-test to the enclosing test (at least, nothing I’ve found) until it completes.
This is the case with all test in general I think. You can give `-V` or `-VV` options to the outer or inner `ctest` command to get it to print more output as it is generated.