Selection of targets from a superbuild's sub-projects

Usage
1

Hi,

TL;DR; Is there a way of selecting specific targets (at build time) from the sub-projects that are built as part of a superbuild?

Consider the simple scenario where there is a superbuild that builds two sub-projects. The reason that those two sub-projects exist is because they are built very differently (e.g. different toolchains).

Each sub-project produces two executables, so let’s call them:
ProjectA: A_exe1, A_exe2
ProjectB: B_exe1, B_exe2

When we run cmake to configure the top-level superbuild, cmake gives us a superbuild configuration that has the two sub-project targets (along with the usual support targets). E.g. running in the “build” directory:
cmake --build . --target help
would list “ProjectA” and “ProjectB” (along with “clean”, etc.)

If we then build those sub-project targets they will (by default) build the “all” target of the sub-project. (E.g. cmake --build . --target ProjectA)

However, we may want to build just a selection of the targets within a sub-project (e.g. some configuration options may be invalid for some of their targets, or we simply don’t want to waste time building them).

We could pass flags “down” to the sub-projects’ during the initial superbuild configuration to “switch stuff on/off”; however, that has drawbacks; e.g.

  • It requires a re-run of the cmake configure, but we do not need reconfiguration; only target selection.
  • It’s invasive on the cmake files of the sub-projects. If we were building these sub-projects stand-alone then we’d just build the specific targets (*).
  • It’s not scalable; we may want a matrix of target combinations from each sub-project (suppose each sub-project can produce 100 targets).

It seems we’d like to do something like:
cmake --build . --target ProjectA:A_exe2 --target ProjectB:B_exe1

I did experiment with first running the cmake configure for the superbuild, and then trying to build the sub-projects from their generated sub-directories. However, the CMakeCache, etc, are not generated for the sub-projects - these are only created at build time, by which time the target is already “all”. E.g. following cmake ../source_root the ProjectA and ProjectB sub-dirs are empty. Perhaps there is a way of forcing the sub-project configuration without the build?

The sub-projects are first-party components, so the source is all there; i.e. it does not appear that the FetchContent approach used for 3PP applies here (because AFAICT that copies the whole source sub-tree).

I’ve had a good search for anything superbuild related but haven’t found anything that appears to fit - apologies if this is made obvious somewhere.

(*) Perhaps the superbuild pattern isn’t suited, and this should just be scripted.

2

Sometimes it is good to know the tools below:

project/build> ninja help
# ...
intro: phony
libCatch2.a: phony
libCatch2Main.a: phony
libfmt.a: phony
libftxui-component.a: phony
libftxui-dom.a: phony
libftxui-screen.a: phony
libsample_library.a: phony
libspdlog.a: phony
sample_library: phony
screen: phony
spdlog: phony
tests: phony
build.ninja: RERUN_CMAKE
clean: CLEAN
help: HELP
bash-5.2$ ninja spdlog
[10/10] Linking CXX static library _deps/spdlog-build/libspdlog.a
bash-5.2$ ninja fmt    
ninja: no work to do.
bash-5.2$ ninja clean
[1/1] Cleaning all built files...
Cleaning... 10 files.
bash-5.2$ ninja fmt 
[3/3] Linking CXX static library _deps/fmt-build/libfmt.a
bash-5.2$ ninja fmt spdlog
[7/7] Linking CXX static library _deps/spdlog-build/libspdlog.a
bash-5.2$ ninja clean
[1/1] Cleaning all built files...
Cleaning... 10 files.
bash-5.2$ ninja fmt spdlog
[10/10] Linking CXX static library _deps/spdlog-build/libspdlog.a
bash-5.2$ ninja clean
[1/1] Cleaning all built files...
Cleaning... 10 files.
bash-5.2$ ninja libCatch2
libCatch2.a      libCatch2Main.a  
bash-5.2$ ninja libCatch2.a
[104/104] Linking CXX static library _deps/catch2-build/src/libCatch2.a
bash-5.2$ ninja intro 
[88/82] Linking CXX static library _deps/ftxui-build/libftxui-component.a
# ...

If you want to try it yourself see cmake_template

3

I’m struggling to see any context in that response that relates to the original post.
This query is about the cmake superbuild pattern; I.e. a top-level project that adds sub-projects using ExternalProject_Add.
The generator used for the build is irrelevant to the question.

4

This project is a sample of a super build project using CPM.cmake and FetchContents:

bash-5.2$ rm -rf out/build/
bash-5.2$ cmake --preset .
CMake Error: No such preset in /Users/clausklein/Workspace/cpp/cmake_template: "."
Available configure presets:

  "debug"         - Debug
  "release"       - Release
  "gcc-debug"     - gcc Debug
  "gcc-release"   - gcc Release
  "clang-debug"   - clang Debug
  "clang-release" - clang Release

bash-5.2$ cmake --preset clang-release | grep -w CPM
-- CPM: Adding package fmt@9.1.0 (9.1.0 at /Users/clausklein/.cache/CPM/fmt/5c4bc51f3df5bb907a0028facff05ba4978aa6e5)
-- CPM: Adding package spdlog@1.11.0 (v1.11.0 at /Users/clausklein/.cache/CPM/spdlog/3f1a872999ca96ce50003a293cd3768892029aef)
-- CPM: Adding package Catch2@3.3.2 (v3.3.2 at /Users/clausklein/.cache/CPM/catch2/8336fa0dd617945aa607363db497172b01dc4bf1)
CMake Deprecation Warning at /Users/clausklein/.cache/CPM/cli11/9e6fd3fcb946a91f4a10e874dc9d09753d96ed7f/CMakeLists.txt:1 
-- CPM: Adding package CLI11@2.3.2 (v2.3.2 at /Users/clausklein/.cache/CPM/cli11/9e6fd3fcb946a91f4a10e874dc9d09753d96ed7f)
-- CPM: Adding package FTXUI@0 (dd6a5d371fd7a3e2937bb579955003c54b727233 at /Users/clausklein/.cache/CPM/ftxui/f0c2f804c00f907a3f5265baa402e97baa4f2ab5)
-- CPM: Adding package tools@ (update_build_system at /Users/clausklein/.cache/CPM/tools/8c8c55a914858ebfc250a90e65f4f4a988a4e20e)
CMake Warning (dev) at CMakeLists.txt:72 (message):
  Building Tests.  Be sure to check out test/constexpr_tests.cpp for
  constexpr testing
This warning is for project developers.  Use -Wno-dev to suppress it.

bash-5.2$ cmake --build --preset clang-release --target help | tail 
libsample_library.a: phony
libspdlog.a: phony
relaxed_constexpr_tests: phony
sample_library: phony
screen: phony
spdlog: phony
tests: phony
build.ninja: RERUN_CMAKE
clean: CLEAN
help: HELP
bash-5.2$
5

As mentioned, the FetchContent approach works well for third-party packages (3PP), because they change rarely. But for nested first-party sub-projects we do not require (nor want) any source code fetching (nor copying).

If those first-party sub-projects are added using ExternalProject_Add then the sub-projects’ targets are not visible in the cmake --build . --target help list (nor make help nor ninja help nor whatever generator…).
The top-level project cannot “see” the sub-projects’ targets.

As an illustration, here’s a simple mock up that provides the scenario as described in the original post.
Superbuild.tar.gz (10 KB)

6

I cant see any good reason to use external_project, except if it is a non CMake build project? You may use fetch_contentorCMP.cmake` and point to a given source dir to build without cloning the subproject.

i.e.: FETCHCONTENT_SOURCE_DIR_<uppercaseName>

If this is set, no download or update steps are performed for the specified content and the <lowercaseName>_SOURCE_DIR variable returned to the caller is pointed at this location. This gives developers a way to have a separate checkout of the content that they can modify freely without interference from the build. The build simply uses that existing source, but it still defines <lowercaseName>_BINARY_DIR to point inside its own build area. Developers are strongly encouraged to use this mechanism rather than editing the sources populated in the default location, as changes to sources in the default location can be lost when content population details are changed by the project.

And CPM.cmake local-package-override

7

Use ninja at build dir and it works:

Claus-iMac:build clausklein$ ninja  -C A help
ninja: Entering directory 'A'
[1/1] All primary targets available:
edit_cache: phony
rebuild_cache: phony
all: phony
build.ninja: RERUN_CMAKE
clean: CLEAN
help: HELP
Claus-iMac:build clausklein$ ninja  -C B help
ninja: Entering directory 'B'
[1/1] All primary targets available:
edit_cache: phony
rebuild_cache: phony
all: phony
build.ninja: RERUN_CMAKE
clean: CLEAN
help: HELP
Claus-iMac:build clausklein$
8

The ExternalProject_Add directive is clearly intended to support cmake projects, after all, that’s what it looks for by default.
However, it’s looking like trying to manage this pattern doesn’t fit, and it’s simply easier to use a script to build the sub-projects independently.

The FetchContent approach appears to always (ultimately) use ‘add_subdirectory’. This isn’t suitable because the sub-projects (in this case) have to be treated as very independent builds, so they do not want pollution of settings from each other.

9

Yes, I agree.

10

I figured out a way of achieving what I need. It’s all there in the docs for ExternalProject, under the Explicit Step Management.
The problem I had essentially boils down to the top-level project not configuring the sub-projects. This means the desired targets of the sub-projects cannot be built because the cmake configuration is not available for them.
This is where the step targets come in. The configurations for those sub-projects can be forced to run by “exposing” the configure step. This is simply just a case of adding STEP_TARGETS configure to the ExternalProject_Add options.
This is then used to force configuration of the sub-project. It means several calls to cmake are required, but it avoids specific shell/platform/generator directives.

So a sequence would be:

cmake ../Superbuild  # Configure top-level project
cmake --build . --target A-configure  # Force configuration of A by executing the "step" target exposed for sub-project A
cmake --build A --target A_exe2  # Now build a specific target of sub-project A
11

Looks like this can be achieved by an explicit execute_process following the ExternalProject_Add (removing the need for the STEP target); e.g.:

execute_process(COMMAND
    ${CMAKE_COMMAND} -S ${CMAKE_SOURCE_DIR}/A -B ${CMAKE_BINARY_DIR}/A
)

I’m surprised there isn’t an option on ExternalProject_Add for forcing just that. Unless I missed it?
( @craig.scott any opinion? :wink:)

12

IMHO: it would be much clearer to write a script, on even better, a GNUmakefile, or not?

13

Sure, as soon as there are multiple commands to type in then a script file might become beneficial.
The aim was that the script could consist purely of calls to cmake. IOW: it serves as not having to remember the cmake command lines, rather than being intrinsically involved in the build logic.

Adding the execute_process entry, the second call to cmake in the above sequence is no longer required; i.e. only two calls to cmake are required in order to build a specific target of a sub-project.