FetchContent in dependency management

I would like to ask for some guidance on best practices and suggestions on how to handle deps in CMake. I am developing CMake-based C++ projects for customers that depend on a couple of libraries but wish to provide a clean out-of-the-box experience for the consumers of these projects. This means, that git clone, configure, build should succeed without end users lifting a finger.

The solution in one case was using Craig Scott’s DownloadProject, which has a crucial shortcoming, which FetchContent is supposed to address (and it does, bur more on that later). The solution in another case using git submodules is pulling in the source code of dependencies even before configuration would start. I also tried using the ExternalProject module in cases, but having to deal with setting up INTERFACE targets for every dep manually is tedious, error prone and not always sufficient, due to builds still missing some files at configure time, sometimes needed as sources, sometimes to decide link language, etc.

My experience is none of these solutions scale. DownloadProject, ExternalProject, FetchContent, git submodules… they all turn into intricate fixups of other project’s mess.

IMHO all of these methods should only be used when the dependency is really part of the same project.

Git submodules are for projects that have multiple… modules, and haven’t transitioned to a monorepo. Same goes for FetchContent and similar CMake modules. They only work reliably when one is in total control of everything pulled in. In all other cases, a singular call to find_package(MyDep REQUIRED) is the only level of concern downstreams should have to deal with. Much like Find Module and Package Config files, these need not be guarded against the targets already existing, because these files are safe for multiple inclusions. The idiom

if(NOT TARGET glm::glm)
  find_package(glm CONFIG REQUIRED)
endif()

is a sad consequence of the new practice of pulling in dependencies into our own build. find_package(glm CONFIG REQUIRED) only conflicts with glm when it’s pulled in as part of the build, because then the ALIAS and INTERFACE targets would clash. What’s really annoying is that every project has to guard it’s own targets too:

if(NOT TARGET MyLib)
  add_library(MyLib ...)
endif()

Because no project knows whether they’re pulled into the build multiple times by a superproject. Moreover, when trying to cater to those both supplying their own deps through APT or Vcpkg, etc and also those enjoying our custom crafted out-of-the-box experience, one has to deal with the subtleties of ALIAS vs. INTERFACE targets, as they have different sets of properties. Here is one of the less complicated ones:

if(NOT DEPENDENCIES_FORCE_DOWNLOAD AND NOT EXISTS "${CMAKE_BINARY_DIR}/_deps/glm-external-src")
  find_package(glm CONFIG)
  # NOTE 1: GLM 0.9.9.0 in Ubuntu 18.04 repo doesn't install the IMPORTED
  #         INTERFACE target, only the legacy variable is defined in glm-config.cmake
  # NOTE 2: auto-fetched subproject build doesn't define the (legacy) variable
  #         anymore, only the INTERFACE target
  #
  # To avoid every test depening on GLM define their deps using
  #
  # add_sample(
  # LIBS
  #   $<$<TARGET_EXISTS:glm::glm>:glm::glm>
  # INCLUDES
  #   $<$<NOT:$<TARGET_EXISTS:glm::glm>>:"${GLM_INCLUDE_DIRS}">
  # )
  #
  # we create the INTERFACE target in case it didn't exist.
  if(glm_FOUND AND NOT TARGET glm::glm)
    add_library(glm::glm INTERFACE)
    target_include_directories(glm::glm INTERFACE "${GLM_INCLUDE_DIRS}")
  endif()
endif()

if(NOT (glm_FOUND OR TARGET glm::glm))
  if(NOT EXISTS "${CMAKE_BINARY_DIR}/_deps/glm-external-src")
    if(DEPENDENCIES_FORCE_DOWNLOAD)
      message(STATUS "DEPENDENCIES_FORCE_DOWNLOAD is ON. Fetching glm.")
    else()
      message(STATUS "Fetching glm.")
    endif()
    message(STATUS "Adding glm subproject: ${CMAKE_BINARY_DIR}/_deps/glm-external-src")
  endif()
  cmake_minimum_required(VERSION 3.11)
  include(FetchContent)
  FetchContent_Declare(
    glm-external
    GIT_REPOSITORY      https://github.com/g-truc/glm
    GIT_TAG             0.9.9.8 # e79109058964d8d18b8a3bb7be7b900be46692ad
  )
  FetchContent_MakeAvailable(glm-external)
endif()

25 lines of script (without comment) for one dep. And I got 5-6-7 deps, each with subtly different annoyances:

  • some abandoned where I have to patch the 8 year old CMake scripts,
  • having to scope their build because they don’t build warning-free, so I have to patch up CMAKE_CXX_FLAGS for the deps only to remove warning-related flags for all supported compilers,
  • My favorite is needing to find the location of headers of the target, because they are shader sources as well and I need to copy next to my executable. Finding the path to those for both the ALIAS and IMPORTED cases I still haven’t figured out.
  • Some projects (SFML) don’t support consuming projects as part of the build (freetype), only in pre-built binary mode using find_package,
  • sometimes other shenanigans.

Bottom line is: I feel like this brave new world of using FetchContent for handling dependencies, and having to self-defend all our scripts against superprojects potentially multiply including them, etc. is a dead end. I am aware that 3.24 has some degree of find_pacakge() integration for FetchContent, but I’m not sure that will resolve this duality. This trend of pulling other people’s stuff into our build either at build or either at configure time has to stop.

(I’m also aware of dependency providers, but that requires mandating minimum of 3.24 (not a short term solution) and also getting all of our customers on board of using package managers. I’d absolutely love the idea of that, but I’m afraid that’s easier said than done. At best I can try to detect from script whether any package manager is being used, and if not, FetchContent Vcpkg and in one go install all the deps during configuration time and use everything through the oneliner find_package interface.)

Would you consider conan out of scope? With the conan cmake integration, the impact on our library consumers was smaller.

1 Like

Sorry for the radio silence on this one. I intend to respond to this properly, which will take some time. I have to go through this for my Professional CMake book updates, but I’m not up to that part yet. I need to prioritise other 3.24 work first.

The 3.24 release significantly expands the landscape of what’s possible in this area, and I want to avoid giving a flippant answer. I’ll come back to this when I can, but it might be a few weeks.

2 Likes

It could be a solution, or something similar for any of the other package managers. (I’m the maintainer of a couple of Vcpkg ports, so I’d sooner wrap that, just due to familiarity.) This is borderline “use a package manager or I’ll use one for you”.

Thank you Scott for the reply. I’ve been a happy user and evangelist of your book (bought it around the 3.16 timeframe, unsure, didn’t keep a copy of the old files). I’ve read the latest docs of FetchContent and I’ve too seen that significant new functionality dropped in 3.24, but all my projects are either very infrastructure HPC stuff (so ultra conservative and still stick to CMake 3.1, so they can compile on custom Commodore64 rigs) or part of gigantic corporate codebases and I don’t get to call the shots on bumping version reqs, even if for my supplied codebases they mean the world. 3.17 is already sunshine and happiness, but 3.24 is still 18-24 months from hitting cmake_minimum_required.

All in all this stop-gap solution of making things part of our build is a dead end, if you ask me. The 7-8 dependencies I tried all bleed for different reasons and I got the scars to prove it.

  • The OpenCL-SDK is one of them. Dependencies.cmake traverses the Dependencies folder, and there’s the zoo of fixups and hosting patches to build scripts to make things work.
  • The other publicly available is rocPRIM, but I haven’t upstreamed the move to FetchContent yet.
    • Yes, another favorite is GCC’s ParallelSTL depending on Intel TBB, but very specific versions. Detecting CMAKE_CXX_COMPILER_ID isn’t enough, because it’s a dep only when using libstdc++ and not libc++, so one needs to check_symbol_exists and if going bulletproof, check for libstdc++ version to know which TBB version to fetch. GCC just has to ruin it for everybody. Why must I have to jump through flaming hoops for the audacity of depending on the STL? (But of course GCC 9, the first to support the ParallelSTL depends on a TBB version that doesn’t build with CMake, just to make it fun having to build it with custom configure/build/install steps.)
    • GCC is the reason I hate using the ParallelSTL, because I have to explain all this garbage to wannabe physicists using Linux, why they must suffer when building sample code. Even though Canonical provides ppa:ubuntu-toolchain-r/test which makes it easy to install new compilers, one can’t use them without getting into CMake deps, minimally this.

@craig.scott
This question is only relevant for v3.24 but…
Do you know if there are any example dependency providers available for vcpkg or conan?
e.g. vcpkg_provide_dependency() , conan_provide_dependency()

Is this subject discussed in your book?
If so, what chapter / section?

I’m not aware of vcpkg or conan having support for the new dependency provider API yet. The API hasn’t been in an official release, so I wouldn’t expect one to be implemented until the 3.24 release has been out for a while (if they choose to support the new API).

I do intend to cover this new feature in the next edition of my book, but until some dependency providers add support for the new API, it will be somewhat limited in what it can demonstrate.