I’m encountering an issue that I don’t know how to solve. With CMake 3.24, --fresh was added as an option that is super helpful when writing iterations of a CMake build.
Even with that option, I’m seeing that SUBBUILD_DIRs keep their same initial state. This has been a particular problem for me. I have a CI job that checks out your branch and runs some checks on it. If I happen to change one of my external_project / fetch_content dependencies to/from URL and GIT_REPOSITORY, I’ll get this error:
Failed to get the hash for HEAD:
fatal: not a git repository: '.git'
It looks like some of this update logic is sticky. I end up having to clean build on the CI whenever these events happen. Is there any way to get around this issue?
Strange, I would expect that switching the download strategy would reset the source directory (like I’ve seen changing from one URL to another does). Maybe there’s a hole somewhere? Can you provide a minimal example with instructions to reproduce so that the behavior can be tracked down in detail?
Here’s a minimal example with instructions to reproduce:
I just walked through those steps to reproduce myself:
~/git/cmake-subbuild-dir-bug main* ❯ cmake -B build 09:15:11 PM
-- CPM: Adding package rapidjson@0 (6089180ecb704cb2b136777798fa1be303618975)
[ 11%] Performing download step (git clone) for 'rapidjson-populate'
-- Avoiding repeated git clone, stamp file is up to date: '/Users/anthony.alayo/git/cmake-subbuild-dir-bug/build/_deps/rapidjson-subbuild/rapidjson-populate-prefix/src/rapidjson-populate-stamp/rapidjson-populate-gitclone-lastrun.txt'
[ 22%] Performing update step for 'rapidjson-populate'
CMake Error at /Users/anthony.alayo/git/cmake-subbuild-dir-bug/build/_deps/rapidjson-subbuild/rapidjson-populate-prefix/tmp/rapidjson-populate-gitupdate.cmake:34 (message):
Failed to get the hash for HEAD:
fatal: not a git repository: '.git'
It looks like going from GIT_REPO to URL is no problem, but going from URL to GIT_REPO causes the error.
There are definitely some robustness holes in FetchContent and ExternalProject not properly handling when some details change. I had many of those fixed at one point, but had to revert them because of regressions in the main feature they were included with. I haven’t had a chance to go back and reapply at least the robustness fixes. I don’t know when I’ll get another chance to do that, sadly.
The --fresh option shouldn’t need to apply for anything to do with FetchContent. If the content you want to download is already present, then there’s nothing that should need to be repeated. The only reason it came up in this discussion is because of the robustness problems in FetchContent’s detection of changing arguments. Making --fresh try to somehow help with that is the wrong solution. The right fix is to address the bugs in the FetchContent detection logic. Those are definitely solvable, as mentioned above.
The past attempts were a mess because of how some but not all ended up needing to be reverted. I recently went through and looked at the code again and thought it looked like it should now be robust. If you want, you can try the latest master branch and see if your example project still exhibits the issue with that. I’m not sure when I’ll next get a chance to do that, so if you’re able to confirm the problem still exists, it will save me some time.
FYI we’re about to cut the release branch for the next release cycle, so it might already be too late to get anything in for that, unfortunately.
Oh, wait, I should have looked more closely. Your problem is a bit different, being about how --fresh interacts with the sub-build used by FetchContent. I didn’t look at that in my recent re-review of the code. It’s unlikely that situation has been fixed. EDIT: I should quite while I’m behind. Ignore the preceding comments, this issue isn’t about --fresh after all.
HOWEVER…
I forgot a very important detail. In the upcoming release, there’s a new CMP0168 policy, and when you set that to NEW, there’s no sub-build at all! The logic is handled directly by the main CMake project. I hadn’t considered whether the --fresh option needed any special handling for that. I’ll need to take a look at whether there’s anything I missed and get back to you. EDIT: There’s nothing for --fresh to do in that case, it’s a separate feature. Please do try out the latest master and see if that has fixed your problem after all.
I just took a very brief look at your demonstrator project just to see what’s there. Can you please try to eliminate the use of CPM and use just FetchContent directly. We want a minimal project that doesn’t depend on other things that could easily be introducing problems rather than CMake itself. I’m not going to be spending time tracing the logic through CPM for this problem.
Can you please try to eliminate the use of CPM and use just FetchContent directly.
Done:
Please do try out the latest master and see if that has fixed your problem after all.
Hopefully I did this right – I built CMake from master and then re-ran the reproduction:
~/git/cmake-subbuild-dir-bug on example-branch ?2 ❯ git checkout main at 06:11:08 PM
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
~/git/cmake-subbuild-dir-bug on main ?2 ❯ ../CMake/bin/cmake -B build at 06:11:15 PM
[ 11%] Performing download step (git clone) for 'rapidjson-populate'
-- Avoiding repeated git clone, stamp file is up to date: '/Users/anthony/git/cmake-subbuild-dir-bug/build/_deps/rapidjson-subbuild/rapidjson-populate-prefix/src/rapidjson-populate-stamp/rapidjson-populate-gitclone-lastrun.txt'
[ 22%] Performing update step for 'rapidjson-populate'
CMake Error at /Users/anthony/git/cmake-subbuild-dir-bug/build/_deps/rapidjson-subbuild/rapidjson-populate-prefix/tmp/rapidjson-populate-gitupdate.cmake:44 (message):
Failed to get the hash for HEAD:
fatal: not a git repository: '.git'
make[2]: *** [rapidjson-populate-prefix/src/rapidjson-populate-stamp/rapidjson-populate-update] Error 1
make[1]: *** [CMakeFiles/rapidjson-populate.dir/all] Error 2
make: *** [all] Error 2
CMake Error at /Users/anthony/git/CMake/Modules/FetchContent.cmake:1907 (message):
Build step for rapidjson failed: 2
Call Stack (most recent call first):
/Users/anthony/git/CMake/Modules/FetchContent.cmake:1609 (__FetchContent_populateSubbuild)
/Users/anthony/git/CMake/Modules/FetchContent.cmake:2134:EVAL:2 (__FetchContent_doPopulation)
/Users/anthony/git/CMake/Modules/FetchContent.cmake:2134 (cmake_language)
/Users/anthony/git/CMake/Modules/FetchContent.cmake:2373 (__FetchContent_Populate)
cmake/RapidJSON.cmake:15 (FetchContent_MakeAvailable)
CMakeLists.txt:11 (include)
-- Configuring incomplete, errors occurred!
Unfortunately it’s still happening on master. I also tried reproducing with the new policy CMP0168 you mentioned. Running with that didn’t fix it either:
~/git/cmake-subbuild-dir-bug on main !1 ?2 ❯ ../CMake/bin/cmake -B build at 06:20:27 PM
-- Policy CMP0168 exists, setting to new
CMake Error at build/_deps/rapidjson-tmp/rapidjson-gitupdate.cmake:44 (message):
Failed to get the hash for HEAD:
fatal: not a git repository: '.git'
Call Stack (most recent call first):
build/_deps/rapidjson-tmp/upload.cmake:7 (include)
/Users/anthony/git/CMake/Modules/FetchContent.cmake:1742 (include)
/Users/anthony/git/CMake/Modules/FetchContent.cmake:1700 (__FetchContent_doStepDirect)
/Users/anthony/git/CMake/Modules/FetchContent.cmake:1607 (__FetchContent_populateDirect)
/Users/anthony/git/CMake/Modules/FetchContent.cmake:2134:EVAL:2 (__FetchContent_doPopulation)
/Users/anthony/git/CMake/Modules/FetchContent.cmake:2134 (cmake_language)
/Users/anthony/git/CMake/Modules/FetchContent.cmake:2373 (__FetchContent_Populate)
cmake/RapidJSON.cmake:15 (FetchContent_MakeAvailable)
CMakeLists.txt:16 (include)
-- Configuring incomplete, errors occurred!