First, a bit of general commentary which gives some background for the broader audience: fetchcontent vs vcpkg, conan? - #3 by craig.scott
TLDR: I think this is a case of choose the right tool for the right job. From what I see, you’re looking for the benefits of a package manager, but without having to actually rely on one.
Your desire to have a simple out-of-the-box experience for your customers that doesn’t require anything more than doing a git clone, configure and build doesn’t leave you with many good choices in this case. I think you’re going to have to compromise somewhere, you’re just going to have to decide where you’re willing to accept that compromise.
If a dependency isn’t well-behaved when incorporated directly into a larger parent build, then FetchContent might not be the right fit for that dependency. The sort of measures you show in your original post are not indicative of what I see in projects that use FetchContent. I think you’re in the unfortunate situation where your dependencies just don’t suit that scenario very well. FetchContent shouldn’t be viewed as the tool for every job. It works well for dependencies and scenarios that meet its requirements. Many do, but quite a few don’t. You need to choose the right strategy for the situation.
It sounds like the dependencies you need to support are not very friendly to being added directly as sources. If they expect to be the top level project, built and installed somewhere for other projects to consume as binaries, then a package manager may be the more suitable path for you. That would allow you to essentially outsource patching and maintaining support for those dependencies. I would recommend against requiring a particular package manager if you can, which means don’t hard-code things like package manager setup files or commands. If you stick to find_package() calls, most package managers will typically be able to handle that, assuming they provide the dependencies your project requires. That would also not lock anyone out of making use of the new CMake 3.24 features if they so wished in their own projects that consume yours.
I’d recommend not trying to do too much for the user like this. Such enforced logic has a habit of getting in the way for some users who might do things you hadn’t considered. Trying to robustly detect whether a package manager is used seems like inviting trouble to me too. Rather than forcing such logic on your consumers, I would advise to stick to conventional workflows and provide clear, concise instructions in your project’s README.md for the common package managers your users may be willing to try.
And since you mentioned that many of your customers are from the HPC space, I’d suggest taking a look at Spack, if you haven’t already. It is quite popular with that user community and is actively supported. It may present a path that (a subset of) your users may be happy with.
Not sure that any of the above is telling you things you didn’t already know. Unfortunately, this seems like one of those cases of “pick your poison”.