How do I get a workspace of independent subprojects

I would like to create a workspace like pattern.
So that I have a directory which contains a CMakeLists.txt and a number of direcotries with other cmake projects an would like to be able to build all of them by configuring from the workspace.
Also I would like to set some things from the common CMakeLists.txt for all the projects.

I don’t yet know what exactly I want to set from the common context. But I would use a common path for custom cmake scripts.

But ist that possible and does it make sense to do this?

1 Like

CMake projects have some level of boilerplate and making them work as both a top-level project and as an add_subdirectory() of another is a little tedious. Things that every project needs to do include setting up install directories, managing global options (e.g., an option to enable sanitizers or coverage), shared vs. static, and some other bits. If you want each project to be able to be built standalone as well as embedded, I would instead recommend using a superbuild approach to build each project standalone, but coordinate their usage together from the top-level. This can be used to pass cache values to each project instead of having to do some big if (is_main_project) block in each.

In my opinion, projects’ CMake should be written in such a way that they fit well into modular superproject structures like this.

What I do is create a top-level β€œWorkspace” repository that contains all the projects as git submodules, and a simple top-level CMakeLists.txt that sets up any needed workspace-wide configuration and adds each subproject with add_subdirectory().

A common use case for me is that each project uses some shared code repository. To make this work modularly, add the shared repo in each project using find_package (or with CPM.cmake, CPMAddPackage). Then at the top level workspace repo, also add the shared code repo as a git submodule and set set (sharedPackageName_DIR "${CMAKE_CURRENT_LIST_DIR}/sharedPackageName"), then each project should reference the shared code in the workspace, so you can edit it as you’re working on the projects and each one will pick up the changes.

Personally, I also create a simple Makefile to the top-level workspace to implement some utility commands, such as committing or updating all the git submodules.

CMake already supports this natively, in each project you can use check PROJECT_IS_TOP_LEVEL

I’m not so worried about the variable name, but the fact that there’s non-trivial amounts of it that needs to go behind such a check.

1 Like

Thanks for your feedback!
I would like to clarify that vast amounts of boilerplate truly are a nuisance to everyone working with the project that does not work inside my workspace, but that it is nothing I am really concerned of.

I intend to build some tooling based on a pattern rendering engine to generate most of my CMakeLists.txt files for me. I am attempting to simplify and unify working with my projects and thought keeping most of the heavy lifting concerning setting up an environment to cmake would be a good idea.

But I did not really thought of why really I would like to do it this way, but I guess my primary intention was that I could export my projects or parts of them, so I could simply find_pakage them in another. Also I would like to centralize everything to a point that I could perform everything about one of my projects from the Workspace level CMakeLists.txt so that my tooling only needs to know here this sits.

Me saying my projects were standalone was maybe not the entirely best way to phrase it. I meant that they are not necessarily depend on each other. They might in parts, but they might as well be completely independent of the rest. Being able to build a projet without a workspace would be the cherry on top.

I took ROS and their management tool as a reference and you can not realistically expect to build a ROS package without catkin helping you, despite cmake being the thing that does the heavy lifting. So I was not really aiming for that, despite it being a sensible thing to do.

My tooling will mostly be for my own convenience tho and not so much for public consumption. I would possibly publish it, if it would turn out less shitty than my last attempts.

I would keep this thread open to hear some more opinions and advice and maybe to ask for help if i get stuck mid way through.

1 Like

This will require a superbuild because find_package won’t work on something you just did an add_subdirectory for. Maybe FetchContent can do some magic here someday, but that’s a question for @craig.scott.

1 Like

I could use a custom script to find the other projects, if it is possible and could make my life somewhat easier.

Also I would ask your opinion, if I missed something in my following thought process:
I would like my tooling to be capable of shifting around larger structures of code, if I want it to. In a way you could think that I would like to automagically break out a library that was internal to the project and now could be used inside another project into its own. So I would prefer that the libraries would be pretty much self contained so this breakout could be abstracted to the following steps:

1.) Copy the directory containing the code down into the workspace.
2.) Generate a new Config for this new project.
3.) Transfer over everything the old project knew about the new one.
4.) Regenerate the top level CMakeLists.txt
5.) Alter the old projects config.
6.) Regenerate the old projects CMakeLists.txt

I was intending to have a directory structure similar to the following tree:

 .
β”œβ”€β”€ cmake
β”œβ”€β”€ CMakeLists.txt
β”œβ”€β”€ ext
β”œβ”€β”€ lib
β”‚   β”œβ”€β”€ CMakeLists.txt
β”‚   β”œβ”€β”€ lib1
β”‚   β”‚   β”œβ”€β”€ CMakeLists.txt
β”‚   β”‚   β”œβ”€β”€ lib1.cpp
β”‚   β”‚   └── lib1.hpp
β”‚   └── lib2
β”‚       β”œβ”€β”€ CMakeLists.txt
β”‚       β”œβ”€β”€ lib2.cpp
β”‚       └── lib2.hpp
β”œβ”€β”€ src
β”‚   β”œβ”€β”€ client.cpp
β”‚   β”œβ”€β”€ monitor.cpp
β”‚   └── server.cpp
└── test
    β”œβ”€β”€ client.cpp
    β”œβ”€β”€ CMakeLists.txt
    β”œβ”€β”€ lib1
    β”‚   └── lib1.cpp
    β”œβ”€β”€ lib2
    β”‚   └── lib2.cpp
    β”œβ”€β”€ monitor.cpp
    └── server.cpp

I intentionally left out some stuff to make the relevant parts more visible. Also this is my testing project. It implements a pretty simple Client-Server-Pattern with zmq and allows to add a debug monitor as a proxy inbetween both. I found it simpler to include this project, that build a minimal one to show the structure.
So if I would like to break out lib1 into its own project to use it elsewhere, I would generate a directory in my workspace folder bearing the following structure and copy over the contents of the lib1 directories from the old one:

.
β”œβ”€β”€ cmake
β”œβ”€β”€ CMakeLists.txt
β”œβ”€β”€ lib
β”‚   β”œβ”€β”€ CMakeLists.txt
β”‚   β”œβ”€β”€ lib1.cpp
β”‚   └── lib1.hpp
└── test
    β”œβ”€β”€ CMakeLists.txt
    └── lib1.cpp

Then It would regenerate all the CMakeLists.txt files in the new and old project and note everything in its own config files.

Or is there a more elegant way I could be doing it, that I am missing?
And the the main part of why I even was bringing it up in the first place: Is there anything about the superbuild that I would need to keep in mind for that to work?

If I got my understanding right, I will need to make my whole workspace into a superbuild and perform ExternalProject_Add() on all of my projects which by themselves could be superbuilds, so they can work standalone?
Is that what the thing I am trying boils down to?

I think the subprojects should not have to change their code at all.

I myself have a situation similar to this, I’ve got Repo A and Repo B. Repo B calls find_package(RepoA). I want to be able to support any/all of the following scenarios:

  • I am only developing repo B by itself. Support either:

    • Using a system installation of repo A (repoA exports a Cmake package configuration file, so repo B’s find_package call works out of the box)
    • repo A isn’t installed on the system - build it as a subproject by creating a find module that fetches the sources from GitHub and calls add_subdirectory(). Currently in CMake for this to work, repo B would need to provide this find module and add it to its own CMAKE_MODULE_PATH.
  • I am developing repo B and repo A on the same machine (but there’s no superproject). Repo B should reference repo A’s sources, again by, inside a find module, doing an add_subdirectory(). This use case can be supported by Repo B if it provides a find module for repo A that allows the developer to set some sort of REPO_A_SOURCE_DIR variable on the command line.

  • I am developing both repos together in a superproject:
    Repo C
    Repo A | Repo B
    In this case, repo C’s CMakeLists.txt can set the REPO_A_SOURCE_DIR variable before calling add_subdirectory(repoB).

In all cases, repo B can simply do find_package(repoA) in the main flow of its CMakeLists.txt. Repo B can support the use cases of not being in a superproject by providing a FindRepoA.cmake that can be as simple as:

if (REPO_A_SOURCE_DIR AND IS_DIRECTORY "${REPO_A_SOURCE_DIR}")
    add_subdirectory ("${REPO_A_SOURCE_DIR}" 
                      "${CMAKE_CURRENT_BINARY_DIR}/RepoA")

    return()
endif()

include (FetchContent)

FetchContent_Declare (RepoA GIT_REPOSITORY ...)

FetchContent_MakeAvailable (RepoA)

Currently I’ve got my workflow set up this way, where RepoB provides this find module and it is committed to source control. I could imagine a tool that auto-generates find modules like this for every dependency in a superproject…

Either I am to stupid, or there is more to it than I think.
I would like to setup tooling like clang-format, clang-tidy, cppinclude,… in my workspace aka. base CMakeLists.txt. Simply find and include the wrappers is not a problem, but I seem to be unable to think of a way to run these tools on all my projects at once…
I thought of using generator expressions to build the paths to each source file and export it to the next stage lower…
So that I get a list of all my source files to run cppinclude and clang-format against. I would - for simplicity sake - run all of these tools per project, since it would not be necessary in most cases to perform mass refactorings on all projects at once. Plus I would get the oppotunity to provide a nice output on progress, and so on…
Or do you guys think using python scripts inside the workspace to invoke the cmake machinery setup inside the projects would be a better match? I would need an efficient way to keep things in sync between the projects in that case…

catkin for all I know had a way to discover projects and things to export from the projects, called cmake on every project and provided a find package utility. That means that I could potentially do the same on my scripts. I will refuse to have the discovery depend on a specific file, so I would have to rely on other things to collect the necessary info in other ways or generate these files as well.

One of these things at least could be handled by cmake as well. that is exporting of libraries. Maybe you want to hinde some internal libraries/targets from the other projects. catkin as a way to do this with its projects.xml files. But I think I once heard in a talk that cmake can hide targets, functions, etc. when using include or add_subdirectory()… Am I misstaken on this and if not, would I be able to exploit this functionality for my purposes?

For running clang-format et al, I use pre-commit. You can have a pre-commit configuration for each project, and/or the top-level workspace.

Yes, I will have pre-commit hooks to run all configured code conditioning per project. I will still have a top-level CMakeLists.txt for my workspace. It will be used to abstract as much stuff away from python as possible and so that I gain a simple way to preset cache values with the ExternalProject_add()