FetchContent_Populate not working as expected (?)

Code
1

Hello, I’m looking for some help with FetchContent_Populate. I’m not sure if this is an issue with my understanding of the docs or with the tool.

I want to use Populate to pull and extract a .tar.gz archive from our internal repository. I also want to make sure that if someone accidentally deletes the extracted files or folder, that they can be easily recovered.

The docs state that when using FetchContent_Populate with parameters, " No check is made for whether content for <name> has already been populated" - https://cmake.org/cmake/help/v3.25/module/FetchContent.html#command:fetchcontent_populate

However, what I am seeing is that if I delete the source directory and then run the configure step again, the folder is created but not populated. Here is a minimal CMake file:

# NOTE: Intended to be run in script mode with cmake -P
include(FetchContent)
FetchContent_Populate(
  example
  URL        https://getsamplefiles.com/download/tar/sample-1.tar
  SOURCE_DIR example
)

and here is my usage:

$ cmake --version
cmake version 3.26.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$ cmake -P CMakeLists.txt 
-- Populating example
-- Configuring done (0.1s)
-- Generating done (0.0s)
...

$ ls example
sample-1.webp           sample-1_1.webp         sample-5 (1).jpg        sample-5.webp

# So far, so good

$ rm -rf example
$ ls
CMakeLists.txt        example-build           example-subbuild
$ cmake -P CMakeLists.txt 
-- Populating example
-- Configuring done (0.1s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/timont/cmake_test/example-subbuild
[100%] Built target example-populate
$ ls example
# Nothing here

Is there a way to force the behaviour I am looking for? I have been looking in the docs all afternoon to no avail. Sorry if I have missed something.

Thanks to anyone who reads this far :slight_smile:

2

Hey Tim,
I think the combination of FetchContent_Declare and FetchContent_MakeAvailable (which works without issue for things without a CMakeLists.txt) should do what you expect (aka re-download/extract if something was removed):

include(FetchContent)
FetchContent_Declare(
  example
  URL        https://getsamplefiles.com/download/tar/sample-1.tar
  SOURCE_DIR example
)

FetchContent_MakeAvailable(example)
3

Hi Jacob,

Thanks for the suggestion. That was the first approach I tried as it appears to be the recommended one on the docs. However I see the same behaviour. On re-runs the example folder is created but files are not placed in it. Here is my output with this script:

# CMake -S . -B build
include(FetchContent)
FetchContent_Declare(
  example
  URL        https://getsamplefiles.com/download/tar/sample-1.tar
  SOURCE_DIR ${CMAKE_SOURCE_DIR}/example
)
FetchContent_MakeAvailable(example)

Terminal:

$ cmake -S . -B build
...
-- Configuring done (2.7s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/timont/cmake_test/build
$ ls example/
sample-1.webp           sample-1_1.webp         sample-5 (1).jpg        sample-5.webp
$ rm -rf example/
$ cmake -S . -B build
...
-- Configuring done (0.3s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/timont/cmake_test/build
$ ls
CMakeLists.txt  build           example
$ ls example/

From what I can tell if I delete the CMake cache then the DOWNLOAD_DIR which contains the archive then it will redownload and extract as expected, but I would like it to force the extraction to the SOURCE_DIR every time.

4

Hi Tim,

I was able to reproduce your issue and did a quick investigation! I thought I had experience your desired behavior previously but I must have confused it with removing built files.

In any case there is a closed issue explaining the reasoning for this (maybe unexpected) behavior: https://gitlab.kitware.com/cmake/cmake/-/issues/24376

tl;dr: For complex builds with many (100s) of dependencies checking the state of the filesystem would add to much time.

Instead of a detailed check cmake creates a subbuild folder (here example-subbuild alongside the SOURCE_DIR in which it touches certain files for each of the populate steps.

If you want to make sure your archive is re-extracted/re-downloaded you basically have two options:

  • remove the example-subbuild dir after running FetchContent. This would force a re-download, no matter the state of the files and is unfriendly to any web resource you might be using.
  • manually check if the dir|contents exists as expected and force a re-download (by removing the subbuild) if not. A basic version would be if(EXISTS example) ..., a more thorough check could be implemented with file(<HASH> ...)

In the end an important question (because no code is the best code) is if this is really necessary? Why would someone selectively delete something from the build tree? And if that happens (by accident?) would the error message of missing files not be enough of a hint?

5

Hi Jacob,

Thank you for taking the time to look into this :slight_smile: Your answer and the linked issue answers my question.

You last point is completely valid, ultimately these files should probably be hidden away in the build directory. It’s just a quirk of the file structure we’ve been using so far and trying to not change that while adding the FetchContent functionality.

Thanks again for your help.

1 Like