Adding libraries from FetchContent to find_package lookup?

Code
1

I am trying to understand the underlying mechanisms behind dependency management in CMake, particularly with regards to FetchContent. FetchContent can be set up to first call find_package and if that doesn’t find anything compatible, to download the requested library.

Therefore, my project should be able to depend on two libraries that both depend on, let’s say nlohmann_json. First dependency checks for nlohmann, doesn’t find it, fetches it. Second dependency checks, finds it, does nothing - the linkable target is already available.

Except that is not what I am observing:

include (FetchContent)

FetchContent_Declare( json
    GIT_REPOSITORY "https://github.com/nlohmann/json"
    GIT_TAG "v3.12.0"
)

FetchContent_MakeAvailable ( json )

find_package( nlohmann_json 3.12 CONFIG REQUIRED)

This yields:

Could not find a package configuration file provided by "nlohmann_json"
(requested version 3.12.0) with any of the following names:

    nlohmann_jsonConfig.cmake
    nlohmann_json-config.cmake

Add the installation prefix of "nlohmann_json" to CMAKE_PREFIX_PATH or set "nlohmann_json_DIR" to a directory containing one of the above files.  If
"nlohmann_json" provides a separate development package or SDK, be sure it
has been installed.

The find_package lookup will be satisfied when it finds said file, even if said file is empty. nlohmann_json creates this file (alongside the version config file) when configured and it is configured automatically by FetchContent. Therefore, this fixes the problem:

find_package( nlohmann_json 3.12 CONFIG REQUIRED
    HINTS "${CMAKE_BINARY_DIR}/_deps/json"
)

But providing this hint means that the one who’s checking knows where the dependency was downloaded in the first place. So I explored further and found that:

find_package(json 3.12 CONFIG REQUIRED)

succeeds! But in a weird way. json_VERSION is unset and json_DIR points to ${CMAKE_BINARY_DIR}/CMakeFiles/pkgRedirects. I wasn’t able to confirm this 100%, but I suspect this somehow relates to PkgConfig and since the version cannot be properly evaluated, this isn’t the intended way of finding the package.

Okay, let’s get back to the original error message - set the <package>_DIR variable. Since setting any variable won’t survive the current scope and chaining PARENT_SCOPE doesn’t make sense in this context, I’ve tried setting the variable with CACHE PATH "docstring" FORCE parameters and miracuously, it worked.

So the question is: Is this the correct, sane, idiomatic way for the library to behave? Should a library called foo set the cache variable foo_DIR to the place where it puts config files so FetchContent can reliably find it? Are there other mechanisms to ensure the library will be always found by find_package in this context?

2

FetchContent can be set up to first call find_package and if that doesn’t find anything compatible, to download the requested library

That sounds more or less like a good plan, but then in your code example you do it the other way around, so first you do FetchContent_* things and only after that you do find_package().

In general, from what I know, if you are using FetchContent, then you should not be using find_package(), as the targets will be already available for linking (just like you said).

But if you want to have FetchContent as a fallback for find_package(), then what I would expect to see is:

find_package(something CONFIG REQUIRED)

if(NOT something_FOUND)
    include(FetchContent)

    FetchContent_Declare(something
        GIT_REPOSITORY git@github.com:some/thing.git
        GIT_TAG "COMMIT-HASH-OR-TAG-OF-INTEREST"
        #OVERRIDE_FIND_PACKAGE # might need that
    )
    FetchContent_MakeAvailable(something)
endif()

target_link_libraries(${YOUR_LIBRARY_NAME}
    PRIVATE
        some::thing # or `something` or whatever is the target name
)
3

Yeah, you’re correct, you should protect FetchContent with find_package. I wanted to simplify the example where you would need two subprojects, one checks, finds nothing, fetches and the other one checks, finds the library, skips the fetch. So assume the code as minimal reproducible that demonstrates the issue.

The core problem is that by default, find_package will not work after first call to FetchContent, therefore anything that you’re trying to protect from double-fetch will still perform the double fetch.

The problem with FetchContent is that if you have transitive dependencies, they all need to check, otherwise they might break your configuration even if everything is compatible (foo might download nlohmann_json to json-src folder, bar might download it into nlohmann_json-src folder and suddenly, you’re trying to add the same target twice). Or, inversely, they might both download it into the same folder, but they are downloading incompatible versions, meaning one of them will link against a wrong version.

4

I got lost in your explanation very quickly. But maybe it will make more sense to others, so let’s wait for other replies.

What I didn’t understand is why there would be more than one FetchContent for the same dependency in the same project and why would you still try to call find_package() after FetchContent (even though you seem to understand that it should not be so).

As for transitive dependencies - certainly, if a dependency that is resolved with FetchContent has dependencies of its own (and it does not resolve them by itself), then you would need to resolve them yourself beforehand. I have actually made an example project for a similar case recently, where a project resolves png which transitively depends on zlib, so you might want to take a loot at that.

In general, I would definitely recommend using a proper package manager for resolving dependencies, such as vcpkg or Conan.

5

I need to take some lessons on how to explain stuff to ppl, this happens to me way too often :grinning_face:

I was making a toy example to show that a particular thing doesn’t work, let’s demonstrate it on a full example:

Top level CMakeLists.txt

cmake_minimum_required ( VERSION 3.28 )

project (demo)

add_subdirectory(foo)
add_subdirectory(bar)

foo/CMakeLists.txt

cmake_minimum_required( VERSION 3.28 )

project (foo)

find_package ( nlohmann_json 3.12 QUIET )

message ( STATUS "nlohmann_json_FOUND in foo: ${nlohmann_json_FOUND}" )
message ( STATUS "nlohmann_json_VERSION: ${nlohmann_json_VERSION}" )
message ( STATUS "nlohmann_json_DIR: ${nlohmann_json_DIR}" )

if (NOT nlohmann_json_FOUND)
    include (FetchContent)

    FetchContent_Declare( json
        GIT_REPOSITORY "https://github.com/nlohmann/json"
        GIT_TAG "v3.12.0"
    )

    FetchContent_MakeAvailable(json)
endif()

FetchContent_MakeAvailable(json)

message ( STATUS "nlohmann_json fetched in foo" )

add_library (foo STATIC foo.cpp)
target_link_libraries (foo PUBLIC nlohmann_json::nlohmann_json)

bar/CMakeLists.txt

cmake_minimum_required ( VERSION 3.28 )

project (bar)

find_package ( nlohmann_json 3.12 QUIET )

message ( STATUS "nlohmann_json_FOUND in bar: ${nlohmann_json_FOUND}" )
message ( STATUS "nlohmann_json_VERSION: ${nlohmann_json_VERSION}" )
message ( STATUS "nlohmann_json_DIR: ${nlohmann_json_DIR}" )

if (NOT nlohmann_json_FOUND)
    include (FetchContent)

    FetchContent_Declare( json
        GIT_REPOSITORY "https://github.com/nlohmann/json"
        GIT_TAG "v3.12.0"
    )

    FetchContent_MakeAvailable(json)
endif()

add_library (bar STATIC bar.cpp)
target_link_libraries (bar PUBLIC nlohmann_json::nlohmann_json)

And this prints:

-- nlohmann_json_FOUND in foo: 0
-- nlohmann_json_VERSION: 
-- nlohmann_json_DIR: nlohmann_json_DIR-NOTFOUND
-- Using the multi-header code from C:/Users/user/demo/_build/_deps/json-src/include/
-- nlohmann_json_FOUND in bar: 0
-- nlohmann_json_VERSION: 
-- nlohmann_json_DIR: nlohmann_json_DIR-NOTFOUND

As you can see, after FetchContent in foo, find_package in bar failed to find the dependency. That’s why I demonstrated the issue originally by doing FetchContent and then find_package to show that something that ought to work doesn’t work.

As for package managers, I am a) trying to understand how underlying mechanisms work and this is kind of byproduct of the whole rabbit hole I fell into, b) you can’t always use them, c) if this worked, you might not even need them.

6

As you can see, after FetchContent in foo, find_package in bar failed to find the dependency

I thought it was clear by now that find_package() is not meant to be used with/after FetchContent. It is okay-ish if you first do find_package() and then fallback to FetchContent, which is what you did in your latest example of foo/CMakeLists.txt (speaking of which, it has a duplicate FetchContent_MakeAvailable(json) line), but then you are still trying to call find_package() in bar/CMakeLists.txt, even though it is predestined to produce the same result (of not finding the dependency package, as it doesn’t look inside the build folder), and then you do a redundant FetchContent for the same dependency again.

If this dependency is common for several targets, then the find_package()/FetchContent things should happen one level up, so before the first add_subdirectory(). Unless I am missing something, and FetchContent results are not available in subfolders scopes?

As for package managers […] if this worked, you might not even need them

For a couple of dependencies that certainly can be true. But when/if you’ll get to maintain a project with at least 10+ dependencies across all the major desktop/mobile/web platforms, I reckon you’d reconsider this soon enough.

7

Okay, I get that it isn’t supposed to work, but that doesn’t invalidate the question I had at the end of the original post:

Is there anything wrong about setting my_library_DIR as a cache variable in CMakeLists for my_library?

To which, as I now realize, the correct answer is: Yes, it is wrong, because it can shadow an otherwise valid installation of my_library. But I can probably append that path to CMAKE_PREFIX_PATH and set that as a cache variable instead. Since it is a list and since CMake works sequentially, there’s probably no risk in manipulating that one.

I really wanted to avoid topic of package managers - on the projects I am working on, I downright can’t use them, for good reasons. FetchContent is something that I can use.

Your recommendation of fetching transitive dependencies one level up is correct, but you have to be in control of all non-leaf dependencies so you can tell them to not fetch by themselves. Or they need a dedicated switch that can disable fetching. Simply because they are unable to reliably check with find_package.

If FetchContent automatically expanded CMAKE_PREFIX_PATH with build folder for configured dependency, it would solve that problem (and you wouldn’t need to fetch manually at top level, because the dependencies would be able to reliably check).

8

Is there anything wrong about setting my_library_DIR

Admittedly, I haven’t even tried to answer this, even though you explicitly marked it as your question, so apologies for all that (probably unnecessary?) derailment, as you seem to understand all those points already, especially given that your goal is to avoid using package managers.

In that case, I am out of my depths, because I never actually used FetchContent in my real projects, and I am only familiar with its mechanics from a couple of sandbox examples that I did out of curiosity.

In general, providing -Dsomething_DIR/-Dsomething_ROOT/etc during project configuration is how you help Find* modules discover things when you call find_package() without CONFIG (which you might have to do, as there are a lot of projects who do not care enough about creating proper CMake configs for their packages). So it is a quite common practice, and I don’t see anything wrong with that. It is worth to mention that those variables are specific to the actual implementations of particular Find* modules: for example FindZLIB would use ZLIB_ROOT, while some other module would expect something_DIR or some other, so you’d need to make sure which one you are expected to provide (but that you have already discovered, as it seems).

However, when it comes to your particular project example, providing -Dsomething_DIR/-Dsomething_ROOT/etc on project configuration won’t do anything for the very first call of find_package() (given that you’ll try to point them to the location inside the build folder), as the actual package will be fetched only after the first FetchContent, which happens later. And then if your question is whether it is correct to “manually” set something_DIR in cache after the first FetchContent so the next find_package() would then find the package - that I do not know. To me it looks wrong (not because it won’t work, but because it just feels wrong), but you are saying that it works for you, so there is that. And manipulating CMAKE_PREFIX_PATH mid-project into considering locations inside the build folder also seems wrong to me (especially that you are supposed to be using the previous FetchContent results and not try to call find_package() still), but yet again “seems” is not a very strong argument either.

9

Still thanks for the discussion :slight_smile:

A slight correction, -Dsomething_DIR affects find_package even in CONFIG mode.

1 Like
11

Ha, I have been relying on CMAKE_PREFIX_PATH for so long that I’ve never really paid attention to other search paths. Indeed, documentation says that <PackageName>_DIR is searched for the config too, so it does make sense to set it with -D (that is if it already exists and does contain the config).

1 Like
12

You’re missing the FIND_PACKAGE_ARGS keyword in your call to FetchContent_Declare(). That’s what activates the “try find_package() first and fall back to using FetchContent if that doesn’t find anything” behavior.

13

Yes, I mention this in the OP:

However I didn’t use it, because I wanted to demonstrate minimum reproducible for a particular behavior. I assume using FIND_PACKAGE_ARGS in FetchContent_Declare won’t find libraries previously fetched by FetchContent.

14

Only the first FetchContent_Declare() CMake encounters for a particular dependency is used. If FetchContent_Declare() is called later for the same dependency, those declared details are discarded and the first set continue to be the ones retained. When FetchContent_MakeAvailable() is called for a particular dependency and if FIND_PACKAGE_ARGS was included in the first set of declared details, then all subsequent calls to either find_package() or FetchContent_MakeAvailable() for that dependency will end up using the same result as the first call to FetchContent_MakeAvailable(). One of the things the internal implementation does is to add some files to the redirection directory (which is recreated every time CMake is re-executed), and that redirection directory is always checked before anything else. This is what ensures later calls within the same run end up using the same thing as whatever the first call ended up doing.

Sorry if that’s a bit unclear, I’m a bit short on time today.

15

You answer is clear enough, but the behavior you’re describing is very, very undesirable from the user standpoint. Assume following project:

CMakeLists.txt

cmake_minimum_required(VERSION 3.28)

project(demo)

add_subdirectory(foo)
add_subdirectory(bar)

foo/CMakeLists.txt

cmake_minimum_required( VERSION 3.28 )

project (foo)

include (FetchContent)

FetchContent_Declare( nlohmann_json
    GIT_REPOSITORY "https://github.com/nlohmann/json"
    GIT_TAG "v3.12.0"
    FIND_PACKAGE_ARGS 3.12 CONFIG
)

FetchContent_MakeAvailable(nlohmann_json)

add_library (foo STATIC foo.cpp)
target_link_libraries (foo PUBLIC nlohmann_json::nlohmann_json)

bar/CMakeLists.txt

cmake_minimum_required ( VERSION 3.28 )

project (bar)

find_package ( nlohmann_json 2.11 REQUIRED )

add_library (bar STATIC bar.cpp)
target_link_libraries (bar PUBLIC nlohmann_json::nlohmann_json)

This code configures just fine, even though it shouldn’t. Nlohmann follows SameMajorVersion versioning scheme, therefore find_package call for version 2.11 should fail (but it doesn’t). Commenting out the FIND_PACKAGE_ARGS from the FetchContent_Declare removes some of the bookkeeping behavior that you’ve mentioned so the configuration fails, but for the reasons explored earlier in this conversation.

There is a design problem with FetchContent that is exascerbated by the official documentation, which says:

For well-known public projects, the name should generally be the official name of the project.

What does this cause? Assume project depending on A and B with both A and B depending on nlohmann::json. Dependency A fetches nlohmann::json version 2.x using the name json. Dependency B tries to nlohmann::json version 3.x using the same name json, following the advice from the documentation. Configuration succeeds because as you’ve said, the second call to FetchContent_Declare( json ... ) is ignored.

What would user expect? User would expect it to fail, because these two versions are not compatible and the dependency B is now compiling/linking against incompatible version.

Now if I inverted the case - suppose both A and B depend on a compatible version, but use different names for FetchContent_Declare - the configuration will fail, because CMake is trying to add the same target twice.

What would user expect? User expects it to succeed, because both dependencies are using compatible version.

Now if the recommendation of the official documentation said that the name should be <library name>-<library version>, it would be much closer to what the user expects, because at least the case of entirely conflicting versions would fail as per user expectation. With regards to compatible versions, sadly the find_package is the only way to evaluate that (at least as far as I know), since the version compatibility is not a property of a project, but a random config file.

That’s why I spent so much time and effort figuring out how find_package could work in conjunction with FetchContent. Unfortunately, while using FIND_PACKAGE_ARGS makes it seem that find_package works, it really does not, as demonstrated by the example CMakeLists.txt at the beginning of this post.

Disclaimer To be fair, I think I understand the technical reasons of why this happens, I just want to point out that using FIND_PACKAGE_ARGS is less desirable than plain find_package, because it gives more power to the user who’s trying to resolve dependency conflicts.