Can someone give a succinct description of the relationship / differences between FetchContent and ExternalProject? It seems like the former adds external dependencies as though they were embedded with add_subdirectory() and the latter builds/installs the dependencies somewhere and (presumably) points at the _DIR of the resulting XXXConfig.cmake file. But I don’t have a good sense of how to choose. Reading @craig.scott 's post here gives me the impression they have very different use cases, but doesn’t seem to paint the full picture. Can anyone help?
I note the the “Using Dependencies Guide” doesn’t even mention ExternalProject, which gives the impression it’s an outdated approach.
Quote: Some strengths of ExternalProject are also its weaknesses. It allows external project builds to be completely isolated from the main project. This means it can use a different toolchain, target a different platform, use a different build type or even an entirely different build system.
For a dependency on LLVM, I’m beginning to think those are all pure strengths. LLVM is not really set up to be consumed as a subdirectory, and it uses a few questionable CMake practices that otherwise would infect the dependent project. It’s very common that people want to use a Release build of LLVM with assertions enabled when they create a Debug build of a dependent project.
Note that ExternalProject doesn’t mix well with add_library. Generally you want to make a “superbuild” (basically a focused package manager) to build everything as ExternalProjects (including your own). The projects themselves then just use find_package() to find each other and the superbuild “stitches” things together (like a package manager).
Maybe I don’t understand what you’re suggesting, but making a project that builds my project and all of its dependencies as external seems very inconvenient for day-to-day development work on my project, because IIUC in the super-project, rebuilds of my project would not be triggered by changes to its source files, and I couldn’t build just my project without somehow bringing in all those external dependencies.
I was thinking I would conditionally use ExternalProject for (some) dependencies when my project is top-level. Is there a drawback to that approach? If so, how does the arrangement you propose avoid that drawback?
Also, can you confirm my general conclusion that the top-level project always needs to be in control over satisfying all of the transitive dependencies of everything that’s built? That seems like the only way dependency diamonds can ever be resolved reliably when there are version constraints. Have I got that right?
The files for find_package(dep) when using ExternalProject_add(dep) won’t exist until build time. So you cannot use find_package(dep) to load your dependency.
There are techniques for actual development. Look for DEVELOPER_MODE in our common-superbuild. What this does is build all of the dependencies of the specific project and provides a developer-mode.cmake script that can be used to pass as -C developer-mode.cmake to configure an arbitrary build tree as if it were in the superbuild (namely finding dependencies).
Yes. Vendoring without offering controls to “unvendor” is a wonderful way to make packagers’ lives hell because somebody else is going to want that dep someday too. If a superbuild is not suitable, it might be that you can document a list of Conan or vcpkg packages that are needed and use those environments to provide dependencies.
The whole common-superbuild project represents a whole new layer of… something… to learn—I don’t even know what ParaView is yet and it appears to be central—so I’d like to be sure I understand why I’m using it rather than an arrangement I can understand purely in terms of the CMake documentation.
That’s only true if you expect to build the dependencies in the configure step of CMake, right? But you don’t have to: you can build and install your dependencies (not necessarily on the system, you can install them locally) and then have your main project fetch them (using CMAKE_PREFIX_PATH if you installed them locally).
See for instance here, where building the dependencies first and the main project next can look as simple as:
# Run the helper script, installing the dependencies in `./dependencies/install`
cmake -DCMAKE_INSTALL_PREFIX=dependencies/install -Bdependencies/build -Sdependencies
cmake --build dependencies/build
# Build the project, telling `find_package()` where to find dependencies
cmake -DCMAKE_PREFIX_PATH=$(pwd)/dependencies/install -Bbuild -S.
cmake --build build
This is essentially the “DEVELOPER_MODE” I referred to above (in the simple case where CMAKE_PREFIX_PATH is sufficient to find all dependencies…sadly not a general case).
You can probably put it as https://a.literal.url.
I’m offering it as an example of an ExternalProject-using project which implements some of the use cases you seem to be looking for. I don’t think I can suggest it for general usage. Projects which end up using it generally don’t even know because they just find_package() their dependencies and the superbuild wires things up so that its copies get found and used. The projects don’t care whether it is vcpkg, conan, rpm, dpkg, Homebrew, or whatever providing the dependencies. A superbuild is just a focused package manager.
Now that got me curious, because I have never seen a case where that doesn’t work (but I run mostly on Linux). Is it “not a general case” on other platforms, or did you mean on Linux too?
Linux too. This isn’t the thread for it. Search the common-superbuild for superbuild_add_extra_cmake_args or see what helper variables VTK forwards for examples.
In my experience, ExternalProject is best used for building dependencies that do not change frequently, and are not part of the day-to-day development workflow. Semi-static dependencies that you update every few months or more.
But for that particular use case, and the flexibility of being able to build the dependencies exactly like you want to, … ExternalProject is fantastic.
disclaimer: as an early contributor to ExternalProject, my opinions about it are probably not completely unbiased
I cannot explain, as I have not been intimately involved with the CMake project for years now, but … my guess is that it is something that a smaller set of people have used, and nobody has had the time to write up something extensive. It takes lots and lots of time to refine a multi-subproject superbuild to work on many platforms, and writing up how to do that takes even more time.
Also, the focus of the Using Dependencies Guide is for sure Using dependencies, and the SuperBuild using ExternalProject approach is for sure about Building/Installing dependencies, and then following the find_package advice of the Using Dependencies Guide to find the now-installed-via-previous-super-build libraries and tools.
Everybody needs to use dependencies, unless they are lucky enough to have a low-level pure project that has zero deps. Fewer people need to build+install their own with all the pre-packaged stuff available these days.
Having said all that, it does seem like ExternalProject should at least be mentioned in the using deps guide.
Happy to share more about my experience with super builds if you want to chat sometime.
I looked. It seems like the doc describing DEVELOPER_MODE misuses the word “dependent” where it means “dependency”; the project can’t possibly know all of its dependents, right?
I appreciate the kind offer! Looking at the docs this restriction worries me. Some people really want to use IDEs and to easily access their debuggers, and I don’t blame them. It’s hard to imagine advantages of the superbuild outweighing this limitation for my project. Also I wonder if the stated primary purpose of superbuilds, “to build various projects into a single prefix as a prepatory step for packaging,” is a major concern for my project. But maybe we can clear that up in our chat.
At least Visual Studio supports loading Ninja-generator build trees these days. Xcode is still Xcode. Note that this is really a common-superbuild restriction as some projects change filenames based on configuration (particularly on Windows) and various places just don’t account for it. See this code.
