Share FetchContent between projects

Our project has a few large Git LFS repositories (some of which are ~10 GB).

We have a few main repositories (repos) that we build different sets of applications of using CMake as the build system. These main repos all depend on the Git LFS repositories. Ideally our main repos would call out what version of the dependent LFS repo they depend on and provide the ability to checkout the right version of the LFS repo as needed.

I’ve tried to setup multiple main repos (or even same checkouts of the same main repo) to shared similar declarations of FetchContent_Declare, pointing to the same SOURCE_DIR. However, when each project configures, they still try to re-clone the repository, even if the SOURCE_DIR is already checked out and at the same commit already points to the branch.

FetchContent_Declare(
    foo
    GIT_REPOSITORY n     ssh://foo-url
    GIT_TAG              1.0.1
    GIT_SHALLOW          TRUE
    SOURCE_DIR           $ENV{HOME}/foo
)

FetchContent_MakeAvailable(foo)

I’ve found that I can accomplish by pointing the main repos to use a shared FETCHCONTENT_BASE_DIR. However, this means the -build and -subbuild directories are also shared. This may not be an issue, but felt a little weird. Do I have any other options here?

I think there’s an issue (or Discourse thread) discussing this, but a shared store for git repositories to avoid nuking and cloning on the slightest change would be beneficial (git worktree would probably be the preferred solution instead).

@craig.scott Do you remember where we discussed this?

1 Like

I wasn’t able to find past discussions in gitlab or here in the forum which talk specifically about git worktrees. I also vaguely recall there has been some mention of it before though.

I’ve resisted adding this sort of functionality (a central cache) so far to FetchContent, as I’ve seen that as more the domain of a dedicated package manager. CPM is one such example which provides this functionality (it is built on top of FetchContent). I’m not questioning the validity of the use case, and I continue to re-evaluate my position on issues like this. I certainly see how it could be useful.

It may be hard to see the connection, but the feature I proposed in https://gitlab.kitware.com/cmake/cmake/-/issues/22688 might also be another route to avoiding multiple downloads of repos and making this area more efficient. It would effectively allow you to have your own directory where you have a set of repos cloned into, then use the redirection to create git worktrees in the build directories that need to use them. Neither the URL redirection functionality or use of git worktrees for clones is implemented in CMake/FetchContent right now though, I only mention it as a way forward that to me seems more aligned with CMake’s overall scope of responsibilities.

1 Like

Thank you for the response.

In the meantime, I will try to leverage CPM with its CPM_SOURCE_CACHE option.

The future issue sounds promising and interesting, too.