Built in package manager for CMake modules

Development
1

Problem Statement

Currently there is no standard way to consume external CMake code.

Which makes it challenging to share CMake code. Which is problematic for organizations, companies, package managers, etc.

Example 1: vcpkg

Lets take vcpkg as an example.

Currently the recommended strategy for consuming vcpkg is to make it a submodule for your project. At one point FetchContent was recommended. But that recommendation was removed:

As I understand the FetchContent recommendation was removed for performance, and FetchContent not working well if used before the first project call (FetchContent / CMAKE_MAKE_PROGRAM issue - #6 by ben.boeckel).

As a result the majority of users consume vcpkg with a submodule. Which isn’t an ideal workflow for everyone.

Example 2: cpm (CMake package manager)

CPM unlike vcpkg only has 1 file with all the logic it needs. This is likely to make it easier to distribute to users.

CPM recommends multiple methods for using it. Either copy pasting the file into your project. Using wget. Or writing some CMake code to grab it for you.

set(CPM_DOWNLOAD_VERSION 0.34.0)

if(CPM_SOURCE_CACHE)
  set(CPM_DOWNLOAD_LOCATION "${CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
elseif(DEFINED ENV{CPM_SOURCE_CACHE})
  set(CPM_DOWNLOAD_LOCATION "$ENV{CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
else()
  set(CPM_DOWNLOAD_LOCATION "${CMAKE_BINARY_DIR}/cmake/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
endif()

if(NOT (EXISTS ${CPM_DOWNLOAD_LOCATION}))
  message(STATUS "Downloading CPM.cmake to ${CPM_DOWNLOAD_LOCATION}")
  file(DOWNLOAD
       https://github.com/TheLartians/CPM.cmake/releases/download/v${CPM_DOWNLOAD_VERSION}/CPM.cmake
       ${CPM_DOWNLOAD_LOCATION}
  )
endif()

include(${CPM_DOWNLOAD_LOCATION})

While this can work it’s not ideal for both users of CPM or the author of CPM.

Solution Proposal

A built in CMake package manager that borrows the syntax of FetchContent.

However, it does NOT call ExternalProject under the hood. Because we are not trying to build C++ projects. We are just trying to incorporate CMake modules.

vcpkg

cmake_minimum_required(VERSION 3.XY)

cmake_module_manager(vcpkg
    GIT_REPOSITORY https://github.com/microsoft/vcpkg/
    GIT_TAG 2022.09.27
    GIT_SHALLOW OFF # vcpkg needs full history
)

set(CMAKE_TOOLCHAIN_FILE "${vcpkg_SOURCE_DIR}/scripts/buildsystems/vcpkg.cmake")
project(FOOBAR LANGUAGES "CXX")

cpm

cmake_minimum_required(VERSION 3.XY)

project(FOOBAR LANGUAGES "CXX")

cmake_module_manager(CPM
    GIT_REPOSITORY https://github.com/cpm-cmake/CPM.cmake
    GIT_TAG v0.38.0
    MODULE_PATH cmake/ # Append to CMake module path
)

include(CPM)

CPMAddPackage("gh:fmtlib/fmt#7.1.3")
CPMAddPackage("gh:nlohmann/json@3.10.5")
CPMAddPackage("gh:catchorg/Catch2@3.2.1")

Ideally there is an easy way to cache/delete what cmake_module_manager retrieves for the user. So that CI doesn’t constantly have to fetch the same thing. Which means downloads must be redirectable outside the BINARY_DIR.

Taking inspiration from CPM I think a combo cache/environment variable would address this.

Or perhaps vcpkg’s caching solution is more appropriate:

Thoughts?

2

You are still free to use FetchContent with vcpkg. The main issue was that deleting the build folder also deletes the fetched vcpkg checkout. Your approach doesn’t fix that and I also deem it necessary. If you don’t want to use a git submodule just use FetchContent instead. Nobody is stopping you from doing that (https://cmake.org/cmake/help/latest/module/FetchContent.html#id14).
Also passing SOURCE_DIR might fix the original issue with the FetchContent approach.

3

I’m confused what you mean. I explicitly say that cmake_module_manager should be able to redirect downloads outside the build folder.

Ideally there is an easy way to cache/delete what cmake_module_manager retrieves for the user. So that CI doesn’t constantly have to fetch the same thing. Which means downloads must be redirectable outside the BINARY_DIR.

4

I like this idea. In our organization we are working on a multi-repo project and we are finding the need to have CMake modules that are used in several repos. Currently that means we are just copying the same .cmake files to the CMake directories of each project. I think this module manager would be a good way to solve that problem.

1 Like
5 
include(FetchContent)
FetchContent_Populate(
  vcpkg
  GIT_REPOSITORY https://github.com/microsoft/vcpkg.git
  GIT_TAG        703a8113f42e794534df5dfc5cece5dabcb949d0
  SOURCE_DIR     "${CMAKE_SOURCE_DIR}/vcpkg"
  QUIET
)

list(APPEND CMAKE_PROJECT_TOP_LEVEL_INCLUDES "${vcpkg_SOURCE_DIR}/scripts/buildsystems/vcpkg.cmake")
list(APPEND CMAKE_TRY_COMPILE_PLATFORM_VARIABLES CMAKE_PROJECT_TOP_LEVEL_INCLUDES)

So I don’t see why a new function is necessary. Same can be done with cmake modules etc.

2 Likes
6

The Dependency Providers feature added in CMake 3.24 is the intended way forward for package managers to integrate with CMake. Package managers would provide a file that users can specify in their CMAKE_PROJECT_TOP_LEVEL_INCLUDES variable. This intentionally avoids having to hard-code anything in the project related to any particular package manager to use or not use.

It does work well before the first project call for some use cases. I do this all the time in client projects to bring in a repo that defines common CMake build logic, provides toolchain files, etc. before the first project() call. At the moment, it is less suitable if the repo you want to bring in is large because, as identified in the linked issues, a copy is downloaded in each build directory you use. If you blow away the build directory, you will have to download that repo again the next time you run CMake on your project.

Caching of downloaded things has come up before a few times. I’ve resisted this in the past on the basis of not wanting CMake to grow beyond its primary focus. It shouldn’t attempt to become a package manager in its own right. But recently I’ve been reassessing my position on the caching side specifically. I am cautious about how we might provide some sort of caching, seeing the problems that I’ve encountered with this in my own use of package managers and employing certain patterns with client projects. It is clear that having a cache that sits outside the build directory would solve some increasingly common problems users are facing.

I’ve indeed done exactly that in at least one client project that uses vcpkg. It requires an assumption about where you want to put your vcpkg directory, and you have to think about who/what is going to manage that repo after it is first populated, but it can be made to work if your constraints and assumptions are compatible with that approach. If the caching I mentioned above was implemented, that caching would be a better longer term solution though.

I agree, but my view is even stronger than that. As a general comment, we’d like to avoid introducing yet another new command for specifying the packages/dependencies to bring into a project. We already have find_package() and FetchContent_MakeAvailable(), with the former being the most canonical way to do it. Any new functionality should be able to continue using those. Projects should not need to be updated to take advantage of new ways of providing dependencies. The Dependency Providers feature was developed to carefully follow that principle.

What’s missing at the moment is a way to define details of a dependency in a way that is independent of the implementation used to provide the dependencies. CMake currently provides that capability only for FetchContent via FetchContent_Declare(). Vcpkg uses manifests. Conan has its own approach. CPM defines new commands which use FetchContent under the hood. I tried to capture a unified way of specifying dependencies in issue 22686, but I am not optimistic it will gain the necessary support by the major package managers. Private feedback I’ve received expressed concern that it wouldn’t allow implementation-specific details to be specified. I haven’t pursued that discussion any further due to lack of time, but also from having the will to engage with it sucked out of me by the effort it took to get the dependency-related features finally implemented in CMake 3.24.

7 Likes