This library contains multiple test programs that I can compile and run with ctest. There are about 100 short test programs.
One result of using a C++ modules build process is that each executable that uses ‘moduleA.cppm’ will compile its own copy of moduleA. This means that I have to compile moduleA multiple times, conferring a compute penalty (which I can live with; it’s part of the design of modules). However, it also means that there are many temporary build files created, conferring a storage penalty. It turns out that after building all the tests for this relatively small library, about 18 GB of build files accumulate! With a modules build, building all the tests consumes absurd amount of storage · Issue #132 · sebsjames/maths · GitHub
My question is whether CMake has some kind of semi-cleanup process, that would allow me to keep just the binaries for the many test programs, but get rid of temporary build files as the build processes, keeping the disk usage relatively small?
Correction: it’s 6 GB (gcc) to 8 GB (clang) of temporary files, not 18 GB. Still rather a lot of storage. In my other project (mathplot) the ~100 example programs there generate >50 GB of temporary files!
One way I could fix this would be to create a test framework in which I collect all 100 small programs into a single one, but a) it would be nice not to have to do that work and b) in the mathplot case, I really do want those examples to be separate programs.
This seems to work fine, but it introduces a non-portability issue, which is that it requires that the build system is Unix.
It also, not surprisingly, removes the ability to re-call ninja to re-build, with an error like:
[14:48:36 bclang20] ninja
ninja: error: '/home/seb/src/maths/bclang20/tests/CMakeFiles/range_intersects.dir/CXXDependInfo.json', needed by 'tests/CMakeFiles/range_intersects.dir/range_intersects.cpp.o.modmap', missing and no known rule to make it
Ok, so here’s my own solution. I make a macro called my_add_executable that I can use in place of add_executable. This adds an add_custom_command call which uses the cmake -E remove command for cross-platform temporary file removal.
If we limit the cleanup to just the *.pcm (if building with clang) and *.gcm (if building with gcc) files after build, we free up most of the temporary storage that was used by the modules build process. These are the largest temporary build files. After removing them, you can still re-call ninja to rebuild your executables. Instead of saying “nothing to do” it does of course need to regenerate those .pcm/gcm files, but that’s fine; the aim here is to build test programs once, without using up a lot of storage (which can be an issue when running on a continuous integration virtual machine).
Oh, I thought that this was a constraint of modules; that for each program you have to compile each of the modules anew, because program 1 might use a template from moduleA.cppm with, say, float specified as a template argument whereas program 2 might use moduleA.cppm with double passed to that template.
Is cmake 4.4 able to determine that module A should be compiled with both float and double (in this example)?
I seem to get the same behaviour with cmake 4.4rc2. When I build my ~100 test programs (in the tests subdirectory), the common modules (some examples are vec.cppm, vvec.cppm, mathconst.cppm) are built for each program, and are stored in build/tests/CMakeFiles/program_name.dir/
If you want to test, you can build my maths project from the dev/cmake_no_cleanup branch, where I made the my_add_executable an alias for add_executable (so that the cleanup workaround is not enabled).
You’re adding the source individually to multiple targets. That’s why it gets rebuilt each time.
Add the source once, to one library target, and then link that library target into everything downstream.
When you add the source individually to multiple targets CMake cannot coalesce BMI creation. BMI coallation looks like this:
Target A needs a BMI for file Foo
Target A asks the owner of file Foo if it already has a BMI available
If the owner already has a BMI available, Target A re-uses the existing BMI
If the owner doesn’t have a BMI available, Target A builds a new one
By adding the source individually into dozens of targets, you have made it each its own owner. Now when each wants to know if a BMI is already available, it asks itself the question, and the answer is of course that no BMI has already been built.
Thanks for this help; I hadn’t explored how to build a library from modules code - in fact, I didn’t even know if it was possible. Your explanation of how the BMIs are made available really helps here! I will go ahead and implement this.
I found this page https://cmake.org/cmake/help/latest/manual/cmake-cxxmodules.7.html which has lots of useful information, but it does not have a nice simple example to get me started*. I can’t actually work out the right cmake lines to add_library() for my list of modules and then to ‘link’ that library to my executable target.
Here was what I guessed:
# A library of module code
add_library(testmodules MODULE)
target_sources_modules(testmodules MODULES ${SM_VEC_MODULES} ${SM_INTERVAL_MODULES}) # SM_VEC_MODULES is a list of modules
# sm::interval test
add_executable(interval_intersects interval_intersects.cpp)
target_link_libraries (interval_intersects testmodules)
but this was wrong (I think the add_library lines are ok, but target_link_libraries emitted an error). Any hints would be appreciated!
*In my view, this is a common problem with the cmake documentation - it doesn’t ever seem to contain usage examples. Is that deliberate?
If you want to link some modules together into a library, and then link that library into an executable, you do this:
# Define a library target. Here I called it 'testmodules'
add_library(testmodules SHARED) # not MODULE!
# Add C++ modules source files to the library target
target_sources_modules(testmodules MODULES path/to/moduleA.cppm path/to/moduleB.cppm) # etc. In practice, you might make a var containing the list of modules
# Define an exectuable target, called interval_intersects here, with its source cpp file(s) listed as normal
add_executable(interval_intersects interval_intersects.cpp)
# Link testmodules to interval_intersects as usual:
target_link_libraries (interval_intersects testmodules)
I’d encourage the cmake documentation maintainers to add an example like this to the official cmake docs, along with another example that shows you that you can target_sources_modules directly to your executable. These might seem obvious, but to a user who is not steeped in cmake, it is difficult to work these things out (I was foxed by the fact that add_library(target_name MODULE) is the wrong thing to use with C++ modules).
The best answer to the original question (thanks @vito.gamerini) is “You mitigate the disk usage problem by building a library containing your modules code, so that the modules code is not rebuilt many times”.
To do this, you add the following lines to your CMakeLists.txt to make a library of your modules:
# Define a library target. Here I called it 'testmodules'
add_library(testmodules STATIC) # SHARED or STATIC* but not MODULE!
# Add C++ modules source files to the library target
target_sources_modules(testmodules MODULES
path/to/moduleA.cppm
path/to/moduleB.cppm
# etc
)
Then for each executable, you link the library in the normal cmake way:
add_executable(prog1 prog1.cpp)
target_link_libraries(prog1 PRIVATE testmodules) # PRIVATE, PUBLIC or INTERFACE is optional, I think PRIVATE is the right choice here
*I found that if I made a SHARED library it would work fine on a Linux platform, but on Windows/Visual Studio, the library would be named testmodules.dll but then the target_link_libraries line would cause Visual Studio to attempt to find the non-existent file testmodules.lib. Asking for a static library resolved this.
You’re mixing concepts which aren’t related to modules, which is why the modules page isn’t useful to you. It assumes a high level of familiarity with build system and CMake concepts already, and is only filling in the parts about modules.
add_library(MODULE) is documented in the add_library() docs and the CMake Buiildsystem docs. Module libraries are runtime-loaded shared objects, intended for use with dlopen. They cannot be linked.
This is completely separate from and predates by decades the C++ language-level concept of a “module”.
Shared libraries have visibility requirements. On Unix-like systems, the default is all symbols are visible, on Windows the default is all symbols are hidden. You need to use toolchain-specific attributes to control symbol visibility. CMake has nothing to do with this, although it offers some helpers for controlling the global visibility like <LANG>_VISIBILITY_PRESET, and WINDOWS_EXPORT_ALL_SYMBOLS.
C++ modules and CMake’s interaction with them change nothing about how C++ toolchains handle symbol visibility.
So there’s nothing C++ module-specific here. You absolutely can use C++ modules in SHARED and MODULE targets. You cannot use SHARED or MODULE targets, regardless of C++ modules or anything else, in the ways you used them here.
For completeness, if you’re wanting to use C++ modules with shared libraries, I’ll refer you to the following article: C++20 Modules, CMake, And Shared Libraries. It goes into details of why C++20 modules and shared libraries are a problematic combination. That article is a couple of years old, but I believe the fundamental problems it highlights still exist, although folks are working on solving them.
You’re mixing concepts which aren’t related to modules
Yes, you’re quite right. The problem is that, as a developer, I spend most of my time writing code in the language, only slightly reluctantly working on the build-toolchain stuff when I have to. Must try harder!
Here, for my add_library(), I just stuck in first MODULE (as you know, that didn’t work because I wasn’t linking at runtime) and then SHARED, which worked on my dev environment (Linux), but then when I ran it through CI, which includes a Windows build, I got an error. I admit that rather than actually try to understand the problem fully, I just hacked in STATIC, and bingo, it built on both platforms.
It assumes a high level of familiarity with build system and CMake concepts already
If I’m not the target readership for those reference pages, I don’t know who is. I need to understand cmake’s commands to use them to build my programs. All I’m saying is that reference pages with some examples are, in my humble opinion, much more useful than those without examples. cppreference.com is getting better for this. It really helps. You’d probably get fewer dopey posts on here from people like me!
Thanks for the pointers to symbol visibility (what black magic is this symbol you talk of?!). I’ll do some testing, make sure I can link a dynamic library to my test programs on the WIndows CI (though static linking is just fine in this case, I think)
Thanks for the article link. I’ll take a read. I’ve had a few issues with dynamic linking against a modules program, especially libstdc++ compatibility when trying to build libstdc++ as a module (and using import std; in my code).
In my maths library case, I think static linking is fine - there is no installed library; the library built here is used to link multiple unit tests and avoid multiple compiles of all the individual modules from the library.
Updated/corrected best answer to the original question (thanks @vito.gamerini) is “You mitigate the disk usage problem by building a library containing your C++ modules code, so that the modules are not rebuilt many times”.
To do this, you add the following lines to your CMakeLists.txt to make a library of your modules:
# Define a library target called 'testmodules', requesting static linking
add_library(testmodules STATIC)
# Add C++ modules source files to the library target
target_sources_modules(testmodules MODULES
path/to/moduleA.cppm
path/to/moduleB.cppm
# etc
)
Then for each executable, you link the library in the normal cmake way: