FetchContent base directory and CMakePresets

Code
, ,
1

I’m using FetchContent to manage third-party dependencies, and I ran into an issue when trying to deduplicate the sub-builds generated for different build configurations. Below is a minimal working example that replicates the issue:

cmake_minimum_required(VERSION 3.21)

# Add application
add_executable(FetchContentTest main.cpp)

include(FetchContent)

# Configure FetchContent
set(FETCHCONTENT_QUIET off)
set(BUILD_SHARED_LIBS ON)

# Cache original FC base dir
set(ORIGINAL_FC_DIR ${FETCHCONTENT_BASE_DIR})

# Redirect FC base dir to a shared location (instead of per-preset)
get_filename_component(FC_NEW_BASE_DIR "../FetchContent" REALPATH BASE_DIR "${CMAKE_BINARY_DIR}")
set(FETCHCONTENT_BASE_DIR ${FC_NEW_BASE_DIR})

FetchContent_Declare(
  sdl2
  URL https://github.com/libsdl-org/SDL/releases/download/release-2.24.0/SDL2-2.24.0.zip
)

FetchContent_GetProperties(sdl2)
if(NOT sdl2_POPULATED)
    FetchContent_Populate(sdl2)

    # Source stays in shared folder, while the build goes to the binary folder
    set(SDL_BINARY_DIR "${CMAKE_BINARY_DIR}/_deps/sdl2-build")
    add_subdirectory(${sdl2_SOURCE_DIR} ${SDL_BINARY_DIR})
    
    set(SDL_LIBS SDL2main SDL2)

    target_link_libraries(FetchContentTest PRIVATE ${SDL_LIBS})
endif()

# Reset FC base dir
set(FETCHCONTENT_BASE_DIR ${ORIGINAL_FC_DIR})

This is combined with CMake presets in the following way:

{
  "version": 3,
  "cmakeMinimumRequired": {
    "major": 3,
    "minor": 21,
    "patch": 0
  },
  "configurePresets": [
    {
      "name": "base",
      "hidden": true,
      "condition": {
        "type": "equals",
        "lhs": "${hostSystemName}",
        "rhs": "Windows"
      },
      "generator": "Visual Studio 17 2022",
      "binaryDir": "${sourceDir}/out/build/${presetName}",
      "cacheVariables": {
        "CMAKE_INSTALL_PREFIX": "${sourceDir}/out/install/${presetName}",   
        "CMAKE_CXX_COMPILER": "cl",
        "CMAKE_CONFIGURATION_TYPES": "Debug;Release"
      },
      "warnings": { "dev": false },
      "vendor": {
        "microsoft.com/VisualStudioSettings/CMake/1.0": {
          "hostOS": [ "Windows" ]
        }
      }
    },
    {
      "name": "windows-x86",
      "inherits": "base",
      "architecture": {
        "value": "Win32",
        "strategy": "set"
      }
    },
    {
      "name": "windows-x64",
      "inherits": "base",
      "architecture": {
        "value": "x64",
        "strategy": "set"
      }
    }
  ],
  "buildPresets": [
    {
      "name": "windows-x64-debug",
      "configurePreset": "windows-x64",
      "configuration": "Debug",
      "cleanFirst": false
    },
    {
      "name": "windows-x64-release",
      "configurePreset": "windows-x64",
      "configuration": "Release",
      "cleanFirst": false
    }
  ]
}

The intended result is that FetchContent downloads and unpacks the sources to a shared location, since this is meant to be agnostic to the project build configuration, and the build output is then directed to a subfolder corresponding to the CMake preset (thus we would separate 32 and 64 bit builds, debug from release, etc.)

When I open this in Visual Studio with a clean project (i.e nothing was generated yet), the initial configuration (e.g 32 bit) works correctly, no errors. Once I switch to a different configuration, I get the following error:

1> [CMake] CMake Error: Error: generator platform: x64
1> [CMake] Does not match the platform used previously: Win32
1> [CMake] Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.

This appears to be due to a conflict with the architecture strategy setting in CMakePresets. If I change it from “set” to “external”, I don’t get the error when switching presets, instead CMake just cancels generation.

How can I make the FetchContent source and sub-build location agnostic to the build configuration, assuming it is possible?

2

Do not share a FETCHCONTENT_BASE_DIR between different builds. That shares more than just the downloaded sources, it also shares all the bookkeeping files used internally. Those do not expect to have the source directory changing between runs. In particular, the sub-build set up to do the download won’t tolerate the architecture changing, which is what you’re seeing because the sub-build uses the same generator as the main build.

While I understand the desire to want to avoid duplicating the downloaded content when you have two or more builds that need the same thing, that isn’t something that FetchContent supports very well right now.

2 Likes
3

Would it be feasible to someday add support for this, perhaps a variable like FETCHCONTENT_SOURCE_CACHE_DIR? The logic could be that to determine the source directory for any FetchContent package, if FETCHCONTENT_SOURCE_CACHE_DIR is set, it will use a subdir of that directory, and if not, then a subdir of FETCHCONTENT_BASE_DIR.

If this feature were supported, it might then become feasible for projects to create a source tarball of each dependency that they require a specific/frozen version of, which could then be fetched on demand to this central directory that is shared between all projects – perhaps accomplished by some sort of integration with ExternalData?

Just a bit of spitballing.

1 Like
4

On another note, about the sub-build sharing the generator with the main build – I have run into errors with this before: if I set the FETCHCONTENT_BASE_DIR to somewhere outside the build tree, then run configure, then delete the build tree and run configure with a different generator, I’ll get an error about the generator not matching the one used previously in the FetchContent sub-build. While this is understandable, I wish FetchContent could support this case by detecting the new generator, and deleting the other temporary directories except for the downloaded sources.

1 Like
5

Perhaps, but it is much more complicated than it may at first seem. You have to think about more than just caching the downloaded content, you also have to consider all the book keeping that FetchContent has to do behind the scenes. That’s already quite complex for a single build (and still has bugs I haven’t had time to fix). Multiplying that by adding support for multiple build directories that share a cached download will only further complicate things. I remain cautiously open to the idea, but have no concrete plans to implement it for the medium term. There are quite a few other FetchContent and ExternalProject-related tasks that are a higher priority for me.

I understand the use case, but right now that sounds more the like responsibility of a full blown package manager. You may also be able to achieve this right now by implementing your own dependency provider. Such a provider could be community-maintained outside of CMake.

That’s another case that would involve much more complexity than I’d want FetchContent to have to deal with. A better solution would be having FetchContent not need to invoke a separate sub-build at all and issue commands directly from within the current CMake process. See issue 21703 and merge request 5749 which implemented the idea, but later had to be reverted due mostly to Windows quoting problems. I hope to eventually implement a slightly reduced version of the idea which at least handles the built-in download method implementations (custom commands would still use a sub-build).

2 Likes
6

All perfectly valid points, I absolutely do not intend to propose that FetchContent introduce heaps of potential edge cases and other points of failure, all for the sake of this particular use case. As you pointed out, this may indeed be the scenario where an entire package manager is more appropriate.

Ultimately, having a solution that has some inefficiencies, but is at least stable and functional, is more valuable than an implementation where all feature requests are met, but at the cost of complications. In my case, FetchContent offers a good solution for importing third-party dependencies in a fast and dynamic way, as opposed to having to set up a package manager and/or having to manually install all the dependencies I might need. Since I’m still in a prototyping phase, this is more than good enough, and I can always swap it out for more sophisticated methods later.

I mention all this because it might be worth making it more explicit in the documentation, so users’ expectations and feedback are pointed in the right direction. They shouldn’t try to use it as a full-on package manager, as it’s probably more reliable to have CMake coordinate with dedicated external tools. FetchContent still has plenty of very useful features that one can take advantage of.

1 Like
7

Apologies for digging up this thread after so long. I noticed that the FetchContent documentation has been extended somewhat, and I found some more discussions since I made the original post.

I was curious: has the behavior of FetchContent changed in any way with regards to the original points that were raised? In particular, the section “Populating Content Without Adding It To The Build” made me wonder if it could be useful.

Also, would there be a way to use FetchContent to pre-build dependencies, rather than add them to the same build? So instead of having to download and build each dependency for every configuration, FetchContent could treat the dependencies as “separate projects” (as though I were building them on their own) and only ensures that their own configuration and build steps are done in a way that’s compatible with the settings of my current project?

8

That’s what ExternalProject is for.

1 Like
9

In that case, I don’t understand why ExternalProject has to run at build time and not at configure time, or why ExternalProject and FetchContent are different (although IIRC the latter makes use of the former)

10

If you want to build dependencies at configure time, I would execute the child build at configure time with execute_process.

1 Like
11

I could give that a try. Now the only mystery for me is that, if this is possible, why can’t FetchContent do it this way? Or is the intent with FetchContent to have the dependencies be “internal” to the project, whereas with ExternalProject they are - as the name implies - external?

And with the above in mind, what makes more sense for the use case outlined in the original post? Technically, yes, I am trying to do what is typically done via package managers, but I feel like this is something that ought to be relatively straightforward to accomplish by combining CMake scripts. In essence, I just need the source code to be available somewhere, then I need to configure and build each dependency via CMake, and then finally connect all the results to each other.

12

Why do you have to build them at configure time though? The idea behind FetchContent is to build things alongside your project, so everything is built during the build phase. ExternalProject also downloads the source code at build time (instead of during configure like FetchContent), which can speed up configure but slows down the build.

1 Like
13

Ok, disregard the above, I think I found the real solution to my problem… almost :smiley:

It turns out that providing FetchContent_Declare with a SOURCE_DIR will ensure that the sources always go to a specific directory. Unfortunately, FetchContent will still re-download the sources if I switch configurations (I presume the relevant bookeeping is placed in the sub-build folder?), but at the very least configuring and building all seem to work without any issues, so my biggest hurdle is solved.

With that said, I only need to figure out the following:

  1. How can I get FetchContent to only fetch the provided source if the path at SOURCE_DIR mentioned above does not exist (i.e has never been fetched before)?
  2. Related to 1., if for example I want it to always fetch the latest commit in a repository, how can I get it to perform the check and re-fetch as needed?
  3. If it’s something like a header-only library, and/or does not use CMake and needs custom scripting, how can I prevent the creation of build/sub-build altogether?
  4. While the build and sub-build folders can be an acceptable redundancy (i.e having to re-create them for each configuration), is there any option to further de-duplicate them?

For 1. and 2., the trivial solution is to just manually check the path, check the hash, etc. but I’m wondering if FetchContent already provides a solution.

For 4., I’m thinking I could have the paths diverge based on only the relevant arguments, which in most cases would be the platform, compiler, and the build type (debug/release/etc.) In fact, at the end of the day, I just need to add the include directories, build the binaries (with a matching compiler, build type, etc.), and link them to my code. In other words, I don’t really need the dependencies to be available as editable projects (e.g in a Visual Studio solution), so is there some way to decouple them in that regard?

Then again, I imagine this may conflict with what FetchContent tries to do out-of-the-box. So would this be better to handle via ExternalProject, or a combination of the two?

14

#1 is solved by CPM.cmake and is the primary reason to use CPM over pure FetchContent.

#3, FetchContent should handle this already. If the downloaded source directory doesn’t contain a CMakeLists.txt, then FetchContent will not perform cmake configure/build on that directory.

#4, no, as far as I know this isn’t possible. You’d have to mess with the internals of FetchContent.

1 Like