FetchContent base directory and CMakePresets

I’m using FetchContent to manage third-party dependencies, and I ran into an issue when trying to deduplicate the sub-builds generated for different build configurations. Below is a minimal working example that replicates the issue:

cmake_minimum_required(VERSION 3.21)

# Add application
add_executable(FetchContentTest main.cpp)

include(FetchContent)

# Configure FetchContent
set(FETCHCONTENT_QUIET off)
set(BUILD_SHARED_LIBS ON)

# Cache original FC base dir
set(ORIGINAL_FC_DIR ${FETCHCONTENT_BASE_DIR})

# Redirect FC base dir to a shared location (instead of per-preset)
get_filename_component(FC_NEW_BASE_DIR "../FetchContent" REALPATH BASE_DIR "${CMAKE_BINARY_DIR}")
set(FETCHCONTENT_BASE_DIR ${FC_NEW_BASE_DIR})

FetchContent_Declare(
  sdl2
  URL https://github.com/libsdl-org/SDL/releases/download/release-2.24.0/SDL2-2.24.0.zip
)

FetchContent_GetProperties(sdl2)
if(NOT sdl2_POPULATED)
    FetchContent_Populate(sdl2)

    # Source stays in shared folder, while the build goes to the binary folder
    set(SDL_BINARY_DIR "${CMAKE_BINARY_DIR}/_deps/sdl2-build")
    add_subdirectory(${sdl2_SOURCE_DIR} ${SDL_BINARY_DIR})
    
    set(SDL_LIBS SDL2main SDL2)

    target_link_libraries(FetchContentTest PRIVATE ${SDL_LIBS})
endif()

# Reset FC base dir
set(FETCHCONTENT_BASE_DIR ${ORIGINAL_FC_DIR})

This is combined with CMake presets in the following way:

{
  "version": 3,
  "cmakeMinimumRequired": {
    "major": 3,
    "minor": 21,
    "patch": 0
  },
  "configurePresets": [
    {
      "name": "base",
      "hidden": true,
      "condition": {
        "type": "equals",
        "lhs": "${hostSystemName}",
        "rhs": "Windows"
      },
      "generator": "Visual Studio 17 2022",
      "binaryDir": "${sourceDir}/out/build/${presetName}",
      "cacheVariables": {
        "CMAKE_INSTALL_PREFIX": "${sourceDir}/out/install/${presetName}",   
        "CMAKE_CXX_COMPILER": "cl",
        "CMAKE_CONFIGURATION_TYPES": "Debug;Release"
      },
      "warnings": { "dev": false },
      "vendor": {
        "microsoft.com/VisualStudioSettings/CMake/1.0": {
          "hostOS": [ "Windows" ]
        }
      }
    },
    {
      "name": "windows-x86",
      "inherits": "base",
      "architecture": {
        "value": "Win32",
        "strategy": "set"
      }
    },
    {
      "name": "windows-x64",
      "inherits": "base",
      "architecture": {
        "value": "x64",
        "strategy": "set"
      }
    }
  ],
  "buildPresets": [
    {
      "name": "windows-x64-debug",
      "configurePreset": "windows-x64",
      "configuration": "Debug",
      "cleanFirst": false
    },
    {
      "name": "windows-x64-release",
      "configurePreset": "windows-x64",
      "configuration": "Release",
      "cleanFirst": false
    }
  ]
}

The intended result is that FetchContent downloads and unpacks the sources to a shared location, since this is meant to be agnostic to the project build configuration, and the build output is then directed to a subfolder corresponding to the CMake preset (thus we would separate 32 and 64 bit builds, debug from release, etc.)

When I open this in Visual Studio with a clean project (i.e nothing was generated yet), the initial configuration (e.g 32 bit) works correctly, no errors. Once I switch to a different configuration, I get the following error:

1> [CMake] CMake Error: Error: generator platform: x64
1> [CMake] Does not match the platform used previously: Win32
1> [CMake] Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.

This appears to be due to a conflict with the architecture strategy setting in CMakePresets. If I change it from “set” to “external”, I don’t get the error when switching presets, instead CMake just cancels generation.

How can I make the FetchContent source and sub-build location agnostic to the build configuration, assuming it is possible?

Do not share a FETCHCONTENT_BASE_DIR between different builds. That shares more than just the downloaded sources, it also shares all the bookkeeping files used internally. Those do not expect to have the source directory changing between runs. In particular, the sub-build set up to do the download won’t tolerate the architecture changing, which is what you’re seeing because the sub-build uses the same generator as the main build.

While I understand the desire to want to avoid duplicating the downloaded content when you have two or more builds that need the same thing, that isn’t something that FetchContent supports very well right now.

2 Likes

Would it be feasible to someday add support for this, perhaps a variable like FETCHCONTENT_SOURCE_CACHE_DIR? The logic could be that to determine the source directory for any FetchContent package, if FETCHCONTENT_SOURCE_CACHE_DIR is set, it will use a subdir of that directory, and if not, then a subdir of FETCHCONTENT_BASE_DIR.

If this feature were supported, it might then become feasible for projects to create a source tarball of each dependency that they require a specific/frozen version of, which could then be fetched on demand to this central directory that is shared between all projects – perhaps accomplished by some sort of integration with ExternalData?

Just a bit of spitballing.

1 Like

On another note, about the sub-build sharing the generator with the main build – I have run into errors with this before: if I set the FETCHCONTENT_BASE_DIR to somewhere outside the build tree, then run configure, then delete the build tree and run configure with a different generator, I’ll get an error about the generator not matching the one used previously in the FetchContent sub-build. While this is understandable, I wish FetchContent could support this case by detecting the new generator, and deleting the other temporary directories except for the downloaded sources.

1 Like

Perhaps, but it is much more complicated than it may at first seem. You have to think about more than just caching the downloaded content, you also have to consider all the book keeping that FetchContent has to do behind the scenes. That’s already quite complex for a single build (and still has bugs I haven’t had time to fix). Multiplying that by adding support for multiple build directories that share a cached download will only further complicate things. I remain cautiously open to the idea, but have no concrete plans to implement it for the medium term. There are quite a few other FetchContent and ExternalProject-related tasks that are a higher priority for me.

I understand the use case, but right now that sounds more the like responsibility of a full blown package manager. You may also be able to achieve this right now by implementing your own dependency provider. Such a provider could be community-maintained outside of CMake.

That’s another case that would involve much more complexity than I’d want FetchContent to have to deal with. A better solution would be having FetchContent not need to invoke a separate sub-build at all and issue commands directly from within the current CMake process. See issue 21703 and merge request 5749 which implemented the idea, but later had to be reverted due mostly to Windows quoting problems. I hope to eventually implement a slightly reduced version of the idea which at least handles the built-in download method implementations (custom commands would still use a sub-build).

2 Likes

All perfectly valid points, I absolutely do not intend to propose that FetchContent introduce heaps of potential edge cases and other points of failure, all for the sake of this particular use case. As you pointed out, this may indeed be the scenario where an entire package manager is more appropriate.

Ultimately, having a solution that has some inefficiencies, but is at least stable and functional, is more valuable than an implementation where all feature requests are met, but at the cost of complications. In my case, FetchContent offers a good solution for importing third-party dependencies in a fast and dynamic way, as opposed to having to set up a package manager and/or having to manually install all the dependencies I might need. Since I’m still in a prototyping phase, this is more than good enough, and I can always swap it out for more sophisticated methods later.

I mention all this because it might be worth making it more explicit in the documentation, so users’ expectations and feedback are pointed in the right direction. They shouldn’t try to use it as a full-on package manager, as it’s probably more reliable to have CMake coordinate with dedicated external tools. FetchContent still has plenty of very useful features that one can take advantage of.

1 Like