Unified Fetched Content with FETCHCONTENT_BASE_DIR

[anthonya]

Over in discussion FetchContent base directory and CMakePresets,
@craig.scott said the following:

@craig.scott

Do not share a FETCHCONTENT_BASE_DIR between different builds. That shares more than just the downloaded sources, it also shares all the bookkeeping files used internally. Those do not expect to have the source directory changing between runs. In particular, the sub-build set up to do the download won’t tolerate the architecture changing, which is what you’re seeing because the sub-build uses the same generator as the main build.

While I understand the desire to want to avoid duplicating the downloaded content when you have two or more builds that need the same thing, that isn’t something that FetchContent supports very well right now.

I wanted to revisit this discussion. Is it possible to make this idea something that FetchContent does support well?

For context, this problem also came up in the JS community and was solved with https://pnpm.io/. On the motivation and FAQ pages, Motivation | pnpm and Frequently Asked Questions | pnpm, they walk through their idea of using a global cached directory for dependencies which then hard links into each individual project’s build folder. I personally tried it and it was fantastic.

This problem currently hit me in the CMake world. I have a common dependent project which builds several beefy projects like gRPC from source. All 4 of my projects that use this dependency have to have their own downloaded copy of it, and their own libs built out. Making FETCHCONTENT_BASE_DIR work with a global cache would make this process much more time and space efficient.

[craig.scott]

@anthonya

I wanted to revisit this discussion. Is it possible to make this idea something that FetchContent does support well?

It may be possible. I think about it very often. It’s a pretty complex feature though, as there’s a number of different download methods, you have to tolerate projects that don’t behave like good CMake citizens (e.g. some write to their source directory, which is why hard-linking to a cached version isn’t appropriate for CMake projects), and I’m very wary of scope creep in CMake expanding this into an area better served by its own dedicated package manager.

I wouldn’t focus on FETCHCONTENT_BASE_DIR as a solution. That’s not a path to happiness. I actually wish I’d never exposed that as part of the FetchContent API. It gets too much attention for a use case it was not meant to support, and it encourages unsafe practices such as trying to share the base dir between builds.

[anthonya]

Sounds fair, thanks for the quick response Craig. You mentioned that this may be a better fit for a dedicated package manager, but I purposely moved away from them in favor of FetchContent.

I tried vcpkg, but the ecosystem just isn’t built for scale. It relies on community contributions to keep ports working, and I encountered several scenarios where I had to opt-out in order to move forward on my builds. I made contributions to CMake builds across several OSS projects, and that turned out to be a much more efficient and scaleable process. It allowed me to reduce the hacks needed my side in order to have clean FetchContent code.

[anthonya]

I was thinking about this over the weekend. If one were to add support for it in CMake, what steps would it take to implement it? Perhaps I and/or others could all chip at it until we get there.

[craig.scott]

It will take me some time to compile a list of things to consider and the explanations to go with them. I probably need to do that anyway, but I have to admit, it’s not high on my priority list at the moment.