Organizing multiple CMake projects and dependencies, and what's wrong with linking libraries from CMake build dirs and never installing anything?

[Dennis_Ferron]

First off, I know similar questions have already been answered, but it’s a complex topic, and I have different requirements. The last person to ask this question comprehensively was, if you read the fine print, authoring libraries (plugin/mods) as his work product, to loaded in one exe. I’m consuming libraries, have multiple exe’s, and am interested in controlling disk space usage.

I’m a hobby game developer. Several libraries I use (or may use) across projects are very large. Ogre is 1.5 GB; QT5 is 2.6 GB, for example. My executable projects (game prototypes) consuming these dependencies are often exploratory efforts, so they are numerous and of both long and short project lifetimes. I don’t want to copy all these dependencies into every project that uses them.

I’m working on simplifying my workflow for creating new game projects. Currently I have all my library dependency folders at the same top-level as game executable projects all together under my CLionProjects/ folder. To make sure the libraries are found where I built them, I’ve been setting variables just above find_package, such as “OGRE_DIR” or “LIBXML2_INCLUDE_DIR” + “LIBXML2_LIBRARY” to point to libraries’ cmake-build-debug directories or *Config.cmake files. (It’s only just started to make sense to me, after recent reading, what the difference is between pointing to *Config.cmake scripts or setting variables directly; I arrived at the latter for LIBXML2 because I couldn’t get the former approach to work.)

I recently learned about the “-C foo.cmake” commandline option, which I realized I could use in lieu of hard-coding these path hints in my executable CMakeLists.txt. I created a(nother top level) folder called “Common” and created “Common\Settings.Debug.cmake” and “Common\Settings.Release-Emscripten.cmake” scripts. Then I can just pass these to each new project and target configuration. Settings.Release-Emscripten.cmake also affords a convenient place to set CMAKE_TOOLCHAIN_FILE, instead of passing it on the CMake commandline. (CLion only provides a tiny one-line box for configuring the CMake commandline in Settings, so it’s not very ergonomic to pass many things here. If I can pare things down to ONE “-C …..\Common\Settings..cmake” argument and throw everything else over the fence into that settings script, it makes things much more manageable.)

As implied, so far I’ve been “reaching into” the cmake-build-debug or cmake-build-release-emscripten build folders to get ahold of compilation products for linking, and I guess the headers from the project source dir come along for the ride (just didn’t work with LibXML2 as previously mentioned, and I had to explicitly set its *_INCLUDE_DIR variable). While I suspect I should be using “install” targets, my reasoning is that in a counterfactual world where these libraries were part of one source tree in my executable, they’d be linked this way in the overall build anyhow. I don’t understand what benefit an intermediate install step, copying the files somewhere (in the process using even more disk space), gives me if I’m not referencing them anywhere else on my system outside the walled-garden of my CLionProjects/ folder.

The other reason I have for being leery of “install” is the toolchain environment(s) I’m working with. My Debug build environment is Msys2/MinGW on Windows, with gcc (NOT Visual Studio). Msys2 is awesome but sort of neither fish nor fowl when it comes to library install locations. If detected as “Windows”, is it “C:\Program Files”? (Please, God no.) Is it “/usr/bin”? “opt/somewhere”? Through CLion, maybe the install step goes to another subfolder of or like cmake-build-debug (again begging the question, what’s the difference between this and linking in the build folder directly). Or maybe it does nothing.

Or maybe the install goes to a location that actually overwrites the Msys2 built-in version of a library, and now system packages are breaking all over. This happened to me, twice. Msys2 uses a rolling release model without version pinning or reverting, so fixing the situation means updating, which updates other things, and so on. This in turn causes compilation errors in user projects built from source; you get to play “what did they deprecate/remove/rename/move this time?” and spelunk in mailing lists for projects you wish you didn’t have to know existed. One time I couldn’t build with gcc until JetBrains released a new version of CLion, because MSys2 or MinGW guys had moved the system C++ headers to a new subdirectory CLion didn’t know about.

But I guess you can change the default install directory though? Obviously must be the case, although, to be clear I didn’t know that before. Actually I still don’t know how much control I have versus the library author and how the mechanism works. For some but not all libraries I’ve built from source, the “install” step is as much an afterthought or “here be dragons” for them as it is for me.

All that’s a round-about way of floating the idea, if I must install maybe I can install all the libraries to a subdirectory (split out by Debug, Release-Emscripten, etc.) of the “Common” folder I’ve already created to hold my settings files? Does that sound right?

In the other similar thread I mentioned in my first paragraph, the user was advised to make a subfolder of the executable project itself be the install target of the libraries. However bear in mind, his use-case was multiple libraries (that he wrote) to be loaded by one executable. For my case I really don’t want to run installs of order N libraries times M executable project destinations. That’s hardly better in disk space, and worse logistically, than duplicating dependencies per project.

Another option that occurs to me (and it seems like CLion is halfway there anyway), can I just… install my libraries over my library projects’ own cmake-build-* directories? Like, make the installed .a and .dll files go to where they already are from the build, and the only new thing a copy of the “include/” folders show up under cmake-build-*? That way any custom-logic run during the install step would be run, but it wouldn’t use extra disk space or time copying binary files that already exist in the build location. (I was being a bit coy saying I don’t understand any benefit - I can imagine install steps might include other logic - but I’m not aware of any specific cases my libraries are taking advantage of that.)

[Dennis_Ferron]

So the first problem I’ve run into using a Common/Settings.Foo.cmake file in the way I described above is if the project I’m building isn’t in a top level folder for some reason (such as when it’s in an “examples” subfolder of another project), then the relative paths will be wrong. I could address this by changing them to absolute paths, but this could also be taken as a hint that perhaps these settings really do need to be per-project and not shared globally for all my projects.

Someone suggested using CMakePresets.json. It seems to be for exactly the issue of how do you share a common default configuration without hardcoding it in CMakeLists.txt. That it still has to be a separate file per-project is not longer a mark against once I consider that putting these settings in a common file for all projects doesn’t work as well as I anticipated.

I had some reservations about whether the presets would work for different build type configurations (requiring different settings values, potentially), but it appears they thought of that in the design of the JSON format as it allows conditional settings &etc. In fact it’s so featureful I don’t understand yet how to use it.

Another suggestion was to use Conan to manage the multiple projects and libraries. I was initially hoping to not have to learn yet another configuration metalanguage, but using Conan for this task does seem really promising. I found an excellently written blog post on the subject:
https://jfreeman.dev/blog/2019/05/22/trying-conan-with-modern-cmake:-dependencies/

My neophyte mental model of how this might work is all my dependency libraries feed in to a local instance of the Conan server, and Conan feeds them back out to my projects as needed. It’ll generate or propagate appropriate fooConfig.cmake or Findfoo scripts. I’m guessing this means I don’t have to copy duplicates of the library artifacts into dependencies folders in every project, because those config or module scripts could just as easily tell Find_Package where to go look for the dependencies where they live outside my consuming projects.

I don’t know enough about Conan yet to know whether it will want to copy everything once into some kind of Conan package repository folder, or if it can also source the files straight from the build folders (eliminating one level of file duplication). However even if Conan does gather binary artifacts into a central location, at least that’s only one extra copy and not N extra copies per consumer. I’m also not sure if Conan can pull them from their build folders, or if they must go to install folders, or if Conan itself runs the Install step.

Something I noticed that does concern me is that although Conan supports building packages with Emscripten (Emscripten — conan 1.54.0 documentation), in the post I linked above from John Freeman’s blog he describes the process of importing Conan packages into CMake as involving the step, “Pass conan_paths.cmake as the CMAKE_TOOLCHAIN_FILE to CMake”. What concerns me is that the way I’m currently making an Emscripten build is to set CMAKE_TOOLCHAIN_FILE to something else, Emscripten.cmake. How can it be both?

Finally (and orthogonal to the Conan discussion), a third idea that occurred to me is maybe instead of “super projects” one could create “super libraries”. It occurs to me that if CMake target imports are transitive, what’s to stop me from creating a trivial static library project which exists only to import all of set of the common libraries I want to have in my game projects? Then in each game project I need only find_package(MyGameDevSuperLibrary), and that would transitively import targets SDL, libXML, Ogre, etc. Would that work as I’m imagining it to?

[ClausKlein]

I am not sure if I understand your problem.

I have worked with conan and vcpkg end it may help to get your dependencies ready to use.

Most often I use CPM.cmake to fetch small dependencies without installing it.

But bigger packages like Boost, QT, LLVM, … I prevent to compile myself, this waste only time!
I am working on OSX and use brew to install most packages and tools I use.

You may have a look at cpp_vcpkg_project and cmake_conan_boilerplate_template.

Hope it helps or give you new inperations.

[Dennis_Ferron]

I’m finding it hard to succinctly state the actual problem I’m trying to solve. I keep turning it into a series of meta-problems. Perhaps I can just walk through my thought process behind how I got here.

My direct goal is I want to develop a project skeleton and repeatable series of set-up steps for quickly starting new game projects for game jams. From this also follows the desire to have an Emscripten build configuration, because more people will try the game if they can do so in the browser and not have to install it.

If the game is 3D I want to use Ogre. If the game is 2D I want to use SDL. A 2D game will likely want to load tile maps, for which the defacto standard is the Tiled map editor and data format. A library for loading this format is libTMX, which in turn depends on libXML2. In the process of setting this up, I started to second-guess my previous (ad hoc) practice of setting FOO_DIR variables to hard-coded paths in the CMakeLists.txt above the find_package call.

I didn’t want to relegate my FOO_DIR dependency path settings to the CMake command line, because in the CLion IDE’s user interface there is only a small single-line text box that all of these settings must be crammed into. When setting up a new project, I have to alt-tab to another CLion window to cut and paste these values from an already set-up project. This cut and paste must be done separately for each build configuration.

What I’d prefer here is if either CLion or CMake provided a way to just say “take all the CMake command line arguments from this text file.” The CMake “-C” command to preset cache variables is almost this; if I only have -D arguments then it is sufficient to capture all of the command line arguments I am currently passing.

Once I had that piece of the puzzle, I began to think about whether I could put the script(s) for the “-C” commandline argument in a central location, and just have one default script for all projects. But that might not be the best thing for reasons I previously mentioned.

Even though CMakePresets.json solves the proximate problem of providing these settings, and it can be checked into source control, I’m still thinking about the larger scope problem of managing and organizing all of these libraries I have built from source. (Not just setting FOO_DIR hints for ad-hoc locations, but have a system that works for me for managing them.) That’s why I’ve started looking into Conan, as well.

Thanks for the link to cmake_conan_boilerplate_template. The repository it is part of is similar to what I’m considering doing with what I figure out here - create a Github project that just documents my personal “best practices” for setting up C++ projects.

(I don’t know how much consideration I want to give to vcpkg just because in my career I’ve seen many times Microsoft promote some technology as “the way” to do things, only to five years later promote some completely different technology as the new new way to do everything instead. Whether they succeed in getting everyone on board first has little bearing on how likely that is to happen. It only determines the amount of fracturing of the developer community it will cause if/when they do this.)

Regarding why go to the trouble of building large packages myself: in the environment I’m using (Msys2/MinGW on Windows), I’ve sometimes run into cases where I could only get things to work building dependencies myself. Ogre, for example, “doesn’t like” the MSys2 SDL package; I have to let Ogre use the version of SDL it automatically downloads in its own build tree. I also couldn’t get “libTMX” to build against the MSys2 provided “libXML2”, but it builds fine against libXML2 I built myself.

In other cases it is the opposite, and things do work better if I use the MSys2 package for a dependency. However with rolling releases and lack of version pinning in MSys2 there’s no stable baseline. A system update can change a library out from under me in a way that breaks the build of all my projects compiled against that library.

Furthermore I prefer to compile my projects for C++20. Technically that implies C++ libraries I use should also be (re)compiled with that same C++ language version setting, although in practice it would be difficult to ensure this happens for all libraries I’m linking, and I’m not sure how critical it really is in practice.

The bottom line is I (feel I) need to control on a case-by-case basis whether find_package should find system libraries or detect and prefer ones I build from source. If I integrate Conan into my workflow, it would be to manage these locally built packages moreso than to pull pre-builds off Conan-Center.

I was going to joke at this point that this way of working culminates in “Gentoo on Windows,” and if that existed, I should just switch to that. However it turns out “Gentoo on WSL” does exist. I just don’t know if WSL can be used to make native 2D or 3D games, but the distinction would be moot for Emscripten builds as that’s a cross-compilation environment anyway.

(I had a hell of a time getting Emscripten installed natively on Windows. Not only the manual setup instructions wouldn’t work for me - the Chocolately package for Emscripten fails silently during install on my machine. I was just about to try installing it in WSL, but in the end I did get my native Windows install of Emscripten working by re-trying the manual setup instructions over from scratch. So I still haven’t tried WSL.)

[imikejackson (Mike Jackson)]

CLion does provide a multiline approach. See the screenshot attached. Just click on the “double arrow” icon thing at the end of the line.

[imikejackson (Mike Jackson)]

So we did something similar to what you are wanting. We created a CMake file (DREAM3D_SDK.cmake) and we feed that through a “-DDREAM3D_SDK=/Users/Shared/DREAM3D_SDK” argument to our initial configuration. We then have this bit of code near the top of our CMakeLIsts.txt file:

message(STATUS "********* STARTING DREAM.3D CONFIGURATION ***********************")

if(NOT "${DREAM3D_SDK}" STREQUAL "")

include("${DREAM3D_SDK}/DREAM3D_SDK.cmake")

get_property(DREAM3D_SDK_STATUS_PRINTED GLOBAL PROPERTY DREAM3D_SDK_STATUS_PRINTED)

if(NOT DREAM3D_SDK_STATUS_PRINTED)

set_property(GLOBAL PROPERTY DREAM3D_SDK_STATUS_PRINTED TRUE)

endif()

else()

message(STATUS "You have elected to NOT set a DREAM3D_SDK CMake variable. You will")

message(STATUS "need to point to the various dependencies that DREAM.3D requires for")

message(STATUS "building. Those that are undefined are listed next:")

if(NOT DEFINED HDF5_DIR)

message(STATUS "Set HDF5_DIR variable to the directory where the hdf5-config.cmake is located.")

endif()

if(NOT DEFINED ITK_DIR)

message(STATUS "Set ITK_DIR variable to the directory where the ITKConfig.cmake is located.")

endif()

if(NOT DEFINED Eigen3_DIR)

message(STATUS "Set Eigen3_DIR variable to the directory where the Eigen3Config.cmake is located")

endif()

if(NOT DEFINED Qt5_DIR)

message(STATUS "Set Qt5_DIR variable to the directory where the Qt5Config.cmake is located.")

endif()

if(NOT DEFINED QWT_INSTALL)

message(STATUS "Set QWT_INSTALL variable to the root directory where Qwt is located.")

endif()

if(NOT DEFINED TBB_DIR)

message(STATUS "Set TBB_DIR variable to the root directory where TBB is located.")

endif()

endif()

The actual file that gets read is produced through another script.

For reference, https://github.com/BlueQuartzSoftware/DREAM3D/blob/develop/CMakeLists.txt

We also build up our own SDK from this project: https://github.com/BlueQuartzSoftware/DREAM3DSuperbuild which is starting to re-invent vcpkg to some extent. We found that sandboxing our SDK made it less likely to have to deal with with variations of builds. We wanted our libraries to be built a certain way that worked for us. With modern package managers like vcpkg this is becoming less needed. And if you are NOT worried about cross platform development then our way is overkill. You can see some of the concepts that we came up with way back in 2015 I think? They tend to still work but are being superseded by things like vcpkg (Assuming all your libraries play nice with VCPKG).

[Dennis_Ferron]

I’ve since discovered a possible argument in favor of having a top level Common folder where I install libraries I build from source. (If I’m understanding the docs correctly) the install folder gets added to the list of places for CMake to search for packages. In theory, this means I might not have to explicitly set the libFoo_DIR variables; they might be found anyway if both projects have the same common install prefix.

In practice, I’ve found it not to be reliable to depend on find_package actually finding my dependencies in the install folder if I don’t set the *_DIR variables manually. In two out of three build targets, find_package finds my child and grandchild dependent libraries in my common install folder, whether or not I set libFoo_DIR variables, but in the Emscripten build preset, it does not find them unless I explicitly set both _DIR variables.

Moreover in the two that do succeed to build, it incorrectly finds the grandchild dependency, libXml2, in a system folder, even though all the following are true:

• I set CMAKE_FIND_PACKAGE_PREFER_CONFIG true at the top of CMakeLists.txt in the EXE and library projects.

• I specify “CONFIG” in all my find_package calls.

• I built the TMX library using the version of libXml2 in my install folder.

• My exe project is using the config script from TMX, it’s just in turn ignoring the libXml2 installed right next to it.

• The system version of libXml2 is not the same version as I compiled TMX against.

And despite all this, it just looooves the version of libXml2 in my MinGW folder. (I suspect the Config script provided by the TMX install may be running a hard-coded module-mode find_package for libXml2 that I cannot override.) I can at least convert the situation into a hard error by adding a pre-emptive “find_package(LibXml2 REQUIRED NO_DEFAULT_PATH)” before calling find_package for TMX.

But turning it into an error doesn’t support the original premise of being able to just install everything to a common install folder and have find_package Just Work and respect that set-up. It just gives me an indication when that hasn’t worked as intended.

So it looks like I will still be setting *_DIR variables manually.

[ClausKlein]

Try to use while build and use your packages:

cmake ... -D CMAKE_PREFIX_PATH=$Workspace/cpp/stagedir -D CMAKE_INSTALL_PREFIX=$Workspace/cpp/stagedir

[Dennis_Ferron]

Now that’s interesting. The difference setting CMAKE_PREFIX_PATH makes is that it fixes the problem of the native builds finding the wrong LibXml2.

What does not change with or without CMAKE_PREFIX_PATH is the two native builds succeed using TMX from the Common folder, and the Emscripten build fails to find TMX or LibXml2 without hints.

So it seems I have a separate problem with either the Emscripten configuration itself, or the installs of the previous libraries to the Common\cmake-install-emscripten folder.

Edit: I’ve confirmed the TMX library itself has the same problem. The native builds find LibXml2 in my common folder with CMAKE_PREFIX_PATH set, but the Emscripten build of TMX does not find it.

I ran WinMerge on the three folders and find few differences (aside from the .a files themselves being binary different, obviously). One difference is the Emscripten libxml2-config-version.cmake file is looking for 32-bit while the other two are 64-bit. So I tried adding,

message( FATAL_ERROR “Did we get here? (Emscripten)” )
and
message( FATAL_ERROR “Did we get here? (Native)” )

right at the top of the the libxml2-config-version.cmake files in the release-emscripten and release-mingw install folders respectively. The release-mingw configuration exits with:

“Did we get here? (Native)”

but the release-emscripten build exits with:

Could not find a package configuration file provided by “LibXml2” with any
of the following names:

LibXml2Config.cmake
libxml2-config.cmake

…indicating it’s not even running the libxml2-config-version.cmake file.

But the libxml2-config.cmake file is there, in the same relative place it’s found in the other folders. Frustrating.

2nd Final Edit (hopefully): after eliminating everything else, it was going to have to be something to do with the Emscripten toolchain cmake file. So I read that script and found it sets CMAKE_FIND_ROOT_PATH_MODE_* variables to ONLY.

I didn’t previously understand how to read the output from CMAKE_FIND_DEBUG_MODE (it generates so much output), but I gave it another go and confirmed that it’s only looking in “…emscripten/cache/sysroot/…”

So I tried setting CMAKE_ROOT_PATH in various places (CMakeUserPresets.json and CMakeLists.txt), but to no avail - it still only looks in “…emscripten/cache/sysroot/…” At least now I know what the problem is, but I’m not sure what to try next to fix it.

3rd Edit: It works if CMAKE_ROOT_PATH is an absolute path. My problem at the end of Edit #2, apparently, was using a relative path.

So that’s what seems to do the trick, is to set CMAKE_ROOT_PATH to my common install folder, as long as I use an absolute path. I don’t know whether also setting CMAKE_PREFIX_PATH is redundant.

[ClausKlein]

We use a common stagedir to install all subprojects used to build our project, and all must have the same CMAKE_INSTALL_PREFIX!
(We do not cross compile, so we do not use CMAKE_ROOT_PATH)