# ExternalProject\_Add cache for git repositories
**URL:** https://discourse.cmake.org/t/externalproject-add-cache-for-git-repositories/4218
**Category:** Code
**Created:** [October 6, 2021, 2:53pm UTC](https://discourse.cmake.org/t/externalproject-add-cache-for-git-repositories/4218 "2021-10-06T14:53:40Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@codeling](https://discourse.cmake.org/u/codeling)
#### Post date: [October 6, 2021, 2:53pm UTC](https://discourse.cmake.org/t/externalproject-add-cache-for-git-repositories/4218/1 "2021-10-06T14:53:40Z")
Inspired by [this other topic](https://discourse.cmake.org/t/feature-idea-fetchcontent-allow-configurable-target-src-folder-i-e-allow-single-source-clone-across-build-configurations/4213), I added DOWNLOAD\_DIR to our superbuild’s `ExternalProject_Add` to avoid re-downloading external libraries’ archive files, which works nicely.
The `DOWNLOAD_DIR` however only works to provide caching for archives retrieved via URL sources, according [to ExternalProject\_Add documentation](https://cmake.org/cmake/help/latest/module/ExternalProject.html#command:externalproject_add).
Since we clone a few libraries from git in our superbuild, I was wondering if there was something like a “cache” for git repositories as well? I would imagine something like a setting specifying a local git repository that acts as a cache; which will be copied to the source directory; then the “original” git repo would be added as remote (if not already existing); then the update step could be performed as normal if necessary; this would mean that only commits not in the local repository yet need to be fetched.
The only, kind of hacky, workaround I can think of at the moment is to have some step before the call to ExternalProject\_Add, which, if no source was cloned yet, takes as input the local repository path and copies it in the place where the ExternalProject\_Add’s download step would load it into…
My use case: I have a superbuild which I want to test automatically; and while I want to do the superbuild from scratch (to fully test it), I also want to avoid large bandwidth usage (i.e. avoid re-downloading/cloning the whole git repository of, let’s say VTK).
---
### Author:  [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [October 6, 2021, 5:53pm UTC](https://discourse.cmake.org/t/externalproject-add-cache-for-git-repositories/4218/2 "2021-10-06T17:53:28Z")
I see two potential solutions here.
## `git clone --reference`
This would cause cloning to be something along the lines of `git clone --reference $commondir`. This `$commondir` can then be set up manually to have fetched from any set of repositories one might want to cache. The issue with `ExternalProject` managing this behind the scenes is that operations would need to:
- preferentially operate on `$commondir` (otherwise new objects just get “orphaned” in the srcdir again);
- know about remote names per project;
- wrangle branches and tag conflicts in some meaningful way.
## Per-repository cache
There could be a per-project cache repository as well placed under some scheme in `DOWNLOAD_DIR` to avoid conflicts (a hash of the repo URL seems the sanest here). Using `git worktree` to check out the actual src directory would be the way to go rather than actually cloning in order to keep the cache directory as the “canonical” location for objects as an actual cache (requiring a higher level Git, but it’s probably OK these days). Note that `git worktree prune` would need to be considered in case the srcdir is ever manually deleted outside of EP’s control.
Cc: @craig.scott
---
### Author:  [@craig.scott](https://discourse.cmake.org/u/craig.scott)
#### Post date: [October 6, 2021, 9:05pm UTC](https://discourse.cmake.org/t/externalproject-add-cache-for-git-repositories/4218/3 "2021-10-06T21:05:49Z")
I’ve been thinking for a while that it would be good to somehow support using git worktrees with `ExternalProject` and `FetchContent`. It would be more efficient from a space and network bandwidth point of view. It might be something to consider as part of [this proposal](https://gitlab.kitware.com/cmake/cmake/-/issues/22688) which talks about URL substitutions controlled by the user, but worktrees would need additional design work beyond what that proposal currently covers.
---
### Author:  [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [June 19, 2023, 2:39pm UTC](https://discourse.cmake.org/t/externalproject-add-cache-for-git-repositories/4218/4 "2023-06-19T14:39:18Z")
Note that the linked proposal was closed since `url.insteadOf` supports it for Git repositories. This is still relevant independently.