RFC: inject CMakeLists.txt to add_subdirectory()

[bmcdonnell-fb]

RFC / Proposal / Feature Request

I’d like to be able to optionally specify a file to the add_subdirectory() command, which it will use as CMakeLists.txt, as if it were overlaid into the subdirectory. The file can have any path and name (not necessarily CMakeLists.txt).

Like an “out-of-source build” (such as described in Professional CMake, section 2.2), this capability would introduce the notion of an “out-of-source configuration”, where a CMakeLists.txt file does not necessarily live next to or above the source files it configures.

My use case:

@bmcdonnell-fb

I have a CMake project using the Ninja generator and GNU Arm Embedded Toolchain, with libraries in Git submodules. I’d like to be able to add the libraries using add_subdirectory() or similar, with its scoping features, without needing a CMakeLists.txt in the subdirs. This would allow me to use the repos unmodified from upstream (since they don’t have CMakeLists.txt files), and maintain my own CMakeLists.txt files in the main project.

To date, I’ve been using add_subdirectory() with forks of the library repos, with CMakeLists.txt added in each. Existing project structure is like this:

MyCMakeProject/
├── CMakeLists.txt
├── src/
│   ├── CMakeLists.txt
│   └── main.cpp
├── libs/
│   ├── CMakeLists.txt
│   ├── library1/
│   │   ├── CMakeLists.txt
│   │   └── library1.cpp
│   └── library2/
│       ├── CMakeLists.txt
│       └── library2.cpp

Using the upstream repos means libs/library[12]/CMakeLists.txt files will go away.

From what I’ve read, it seems like my options are:

1. Use include() instead of add_subdirectory(). This means no separate CMake scopes, and everything gets built directly into the main project.
2. Add extra subdirs for the upstream repos, such as shown below, to be able to continue to use add_subdirectory().

MyCMakeProject/
├── CMakeLists.txt
├── src/
│   ├── CMakeLists.txt
│   └── main.cpp
├── libs/
│   ├── CMakeLists.txt
│   ├── library1/
│   │   ├── CMakeLists.txt
│   │   └── upstream-repo/
│   │       └── library1.cpp
│   └── library2/
│       ├── CMakeLists.txt
│       └── upstream-repo/
│           └── library2.cpp

I believe a feature such as I’m suggesting here would allow a flatter, more straight-forward file/dir layout, like this:

MyCMakeProject/
├── CMakeLists.txt
├── src/
│   ├── CMakeLists.txt
│   └── main.cpp
├── libs/
│   ├── CMakeLists.txt
│   ├── cmake/
│   │   ├── CMakeLists_lib1.txt
│   │   └── CMakeLists_lib2.txt
│   ├── library1/
│   │   └── library1.cpp
│   └── library2/
│       └── library2.cpp

libs/CMakeLists.txt would contain something like this:

add_subdirectory(library1 LISTS_FILE cmake/CMakeLists_lib1.txt)
add_subdirectory(library2 LISTS_FILE cmake/CMakeLists_lib2.txt)

libs/cmake/CMakeLists_lib1.txt would then be treated by CMake as if it were libs/library1/CMakeLists.txt, so it contains e.g.

add_library(library1 STATIC)
target_sources(library1 PRIVATE "library1.cpp")

[brad.king]

I’m not in favor of adding this. In general we’ve avoided naming CMakeLists.txt anything else.

[bmcdonnell-fb]

Thanks for your feedback.

How do you propose to use CMake in cases such as I described? Isn’t this common enough to consider?

My example here actually oversimplifies the libraries that I’m probably going to wind up using. They have nested git submodules, so I won’t be able implement the option 2 (extra subdirs).

[brad.king]

CMake itself vendors several third-party libraries (with options to use external copies) and has no problem maintaining CMakeLists.txt files embedded among their source trees. We use scripts to import snapshots of the upstream libraries into pristine vendor branches and then subtree-merge them into our repo, integrating with any local changes such as the addition of CMakeLists.txt.

To date, I’ve been using add_subdirectory() with forks of the library repos

Yes. In the case of Git submodules one can point them at dedicated forks/branches where CMakeLists.txt files are added. That also has the advantage of versioning the CMake build system with the code it builds, and enabling it to be used by multiple dependents without duplication. I consider this better than the solutions proposed above, and it sounds like you’re already doing it.

[bmcdonnell-fb]

@brad.king

CMake itself vendors several third-party libraries (with options to use external copies) and has no problem maintaining CMakeLists.txt files embedded among their source trees. We use scripts to import snapshots of the upstream libraries into pristine vendor branches and then subtree-merge them into our repo, integrating with any local changes such as the addition of CMakeLists.txt.

Interesting. Not a fan of git subtree, personally. I prefer the clearer delineation that git submodule provides. (Also the upstream vendor is using git submodule in my case.)

@brad.king

To date, I’ve been using add_subdirectory() with forks of the library repos

Yes.

Sorry, I don’t follow. You quoted a statement of mine, not a question. What does your “yes” mean here?

@brad.king

In the case of Git submodules one can point them at dedicated forks/branches where CMakeLists.txt files are added. That also has the advantage of versioning the CMake build system with the code it builds, and enabling it to be used by multiple dependents without duplication. I consider this better than the solutions proposed above, and it sounds like you’re already doing it.

For want of the feature I proposed here, I’m working on implementing this as I’m refactoring our library submodules. I’d rather pull the CMakeLists.txt files out of the submodules, so that I don’t have to maintain custom branches within them. But I have several submodules, some nested, so I don’t want to have a single monolithic CMakeLists.txt for them all.

Suppose if I want to only build some of the files in a submodule for one project, and a different subset for another. In this case doesn’t it make sense to allow separation of CMakeLists.txt from the source it configures? CMakeLists.txt gets captured in the project commit, and the submodule commit is pegged.

[brad.king]

@bmcdonnell-fb

What does your “yes” mean here?

“Yes, it’s a good approach.”