Can you combine FetchContent and ExternalProject?

Code
1

A query for the experts. I’m trying to transition building a set of base libraries currently done with GNU Make to CMake. Now, I’m having fairly good luck with ExternalProject_Add, but I have only one issue with that: downloading is done at build time, not configure time. This is only a problem for me because I would like to build these libraries at a supercomputing center on a compute node so that I don’t take up shared head nodes. But compute nodes cannot see the internet, thus any URL fetches will fail.

So, I’d love to be able to use FetchContent to get the needed tarballs at cmake time (maybe for a variety of compiler/MPI stacks) and then get compute nodes and do a ‘make -j10’ on each in the build dirs where ExternalProject takes care of that.

Is this possible? I know at some point a user (say, me) will need to do something on a head node. I’d just prefer it to be the thin Fetch step and not the expensive Build step.

2

Ordinarily, I advise choosing one or the other rather than mixing FetchContent and ExternalProject. More specifically, if you can use FetchContent to not just download the dependencies but also to add them directly to your build, that is typically simpler than ExternalProject (read up on using FetchContent_MakeAvailable() specifically). It relies on the dependency projects not doing things like assuming they are the top level project or using target names that other projects also define, but if you don’t hit either of those problems, then it’s probably the simpler route.

In your situation, it may still make sense to mix the two approaches. You want the sources to be downloaded at configure time, which FetchContent can achieve for you (you can’t use FetchContent_MakeAvailable() then though). If you genuinely do need to build the dependencies as external builds, then ExternalProject is the right tool for that. You can tell ExternalProject to use the source locations that FetchContent will have already populated for you at configure time. In the call to ExternalProject_Add(), omit any download or update details and instead provide SOURCE_DIR, with a value being provided by FetchContent. For example (untested, but hopefully you get the idea):

FetchContent_Declare(someproj
    GIT_REPOSITORY https://github.com/someGroup/someproj
    ...
)
FetchContent_GetProperties(someproj)
if(NOT someproj_POPULATED)
  FetchContent_Populate(someproj)
endif()

ExternalProject_Add(someproj
    SOURCE_DIR ${someproj_SOURCE_DIR}  # Provided by FetchContent
    ...
)

Just to be clear, your approach relies on the head node being identically configured to the compute nodes (same paths, exact same tool versions, etc.).

3

This does indeed work, thanks! Unfortunately, it looks like the CMake in HDF5 and netCDF is currently not quite good enough for me to do a “let’s use CMake as much as we can” version. Time for fun with CONFIGURE_COMMAND