add_subdirectory() for Git Submodules without CMakeLists.txt?

Usage
1

I have a CMake project using the Ninja generator and GNU Arm Embedded Toolchain, with libraries in Git submodules. I’d like to be able to add the libraries using add_subdirectory() or similar, with its scoping features, without needing a CMakeLists.txt in the subdirs. This would allow me to use the repos unmodified from upstream (since they don’t have CMakeLists.txt files), and maintain my own CMakeLists.txt files in the main project.

To date, I’ve been using add_subdirectory() with forks of the library repos, with CMakeLists.txt added in each. Existing project structure is like this:

MyCMakeProject/
├── CMakeLists.txt
├── src/
│   ├── CMakeLists.txt
│   └── main.cpp
├── libs/
│   ├── CMakeLists.txt
│   ├── library1/
│   │   ├── CMakeLists.txt
│   │   └── library1.cpp
│   └── library2/
│       ├── CMakeLists.txt
│       └── library2.cpp

Using the upstream repos means libs/library[12]/CMakeLists.txt files will go away.

From what I’ve read, it seems like my options are:

  1. Use include() instead of add_subdirectory(). This means no separate CMake scopes, and everything gets built directly into the main project.
  2. Add extra subdirs for the upstream repos, such as shown below, to be able to continue to use add_subdirectory().
MyCMakeProject/
├── CMakeLists.txt
├── src/
│   ├── CMakeLists.txt
│   └── main.cpp
├── libs/
│   ├── CMakeLists.txt
│   ├── library1/
│   │   ├── CMakeLists.txt
│   │   └── upstream-repo/
│   │       └── library1.cpp
│   └── library2/
│       ├── CMakeLists.txt
│       └── upstream-repo/
│           └── library2.cpp

Questions:

  1. Is my analysis correct, that these are my options?
  2. Is there a way to provide CMakeLists.txt to add_subdirectory() from outside the subdir?
  3. If not, would developers consider adding such a feature?
2

Another option would be to build your own directory hierarchy just for the CMakeLists.txt files, and set up a mechanism to refer to the upstream dirs. Something like this:

MyCMakeProject/
├── CMakeLists.txt
├── src/
│   ├── CMakeLists.txt
│   └── main.cpp
├── libs/
│   ├── CMakeLists.txt
│   ├── library1/
│   │   └── library1.cpp
│   ├── library2/
│   │   └── library2.cpp
│   └── mycmake/
│       ├── library1/
│       │   └── CMakeLists.txt
│       └── library2/
│           └── CMakeLists.txt

Inside MyCMakeProject/libs/CMakeLists.txt, create a function like this:

function(setup_upstream_source)
  cmake_path(RELATIVE_PATH
    CMAKE_CURRENT_SOURCE_DIR
    BASE_DIRECTORY ${CMAKE_SOURCE_DIR}/libs/mycmake
    OUTPUT_VARIABLE result
  )
  set(UPSTREAM_CURRENT_SOURCE_DIR ${CMAKE_SOURCE_DIR}/libs/${result} PARENT_SCOPE)
endfunction()

Then, inside the CMakeLists under mycmake, call the function at the top and then just use ${UPSTREAM_CURRENT_SOURCE_DIR} to refer to paths within the corresponding upstream project.

3

Thanks for the suggestion.

It turns out I have more custom modifications than I realized in the libraries, so I’m now leaning toward injecting the CMakeLists.txt files along with them. However, suppose for a moment that I don’t do that.

As I initially described, on the surface, it seems like I’d want to be able to add_subdirectory() and pass in a CMakeLists.txt from outside, to save me the trouble of the type of workaround you suggested. (i.e. A possible CMake feature request.) Maybe that wouldn’t work recursively, though?

4

You can use FetchContent to pull down the project from a URL, apply patches (which can add a whole new CMakeLists.txt as well as make changes to existing source files) and then build the code as part of your project.

Another option may be to use vcpkg which provides similar abilities to download source code, apply patches and add a CMake based build where one doesn’t already exist. Then you can leverage additional capabilities from vcpkg like binary caching and so-on. However, I’m uncertain how vcpkg works with cross-compiling since it hijacks the CMAKE_TOOLCHAIN_FILE mechanism to inject dependencies at the time the project function is called.

The FetchContent route would seem to be the smallest change from where you are currently.

5

Thanks for the info. I don’t think these suggestions fit well with my existing workflow, though. Please correct me if I’m wrong.

I don’t want CMake to interfere with or replace the fetching or content management of Git / git submodule. And I want “clean” builds, with the source fully captured in git commits.

A full package manager would be a replacement for git submodules, right?

6

I don’t want CMake to interfere with or replace the fetching or content management of Git / git submodule. And I want “clean” builds, with the source fully captured in git commits.

It’s a different workflow, but you can still do this with FetchContent. I suppose it depends on how actively you are modifying the dependency. If it’s a routine thing, then FetchContent is probably not the best fit, but your own parallel directory structure so that add_subdirectory works as expected, but the sources are pulled from the submodule instead of being in the same directory as the CMakeLists.txt file.

A full package manager would be a replacement for git submodules, right?

Again, it depends on how often you’re modifying the dependencies. Generally speaking both the FetchContent approach and the vcpkg approach assume relatively stable dependencies. Both give you the ability to apply patches and/or add your own CMakeLists.txt for the dependencies.

With FetchContent, the targets are more directly included in your build (e.g. in a Visual Studio solution they appear as targets in the IDE), whereas with vcpkg the coupling is looser and while you can step through the source in the debugger, the dependencies won’t appear as targets in your VS solution.

I just thought I’d toss out these two alternatives as other options for you to consider.

7

Thanks for the additional info.

That’s not really the case for this project. The submodules do require occasional modification–sometimes merely as a one-off for testing (not committed).

I fleshed my idea out a bit in RFC: inject CMakeLists.txt to add_subdirectory().

8

When I need to use libraries that don’t support CMake, I usually make a small wrapper repo that contains the CMakeLists.txt, and that wrapper repo is what the main project includes via FetchContent

1 Like