Here’s a brief outline of my situation:
- I’m working on a large Windows Kernel Mode Driver
- Our unit tests need to run in user mode
- Our toolchain is setup to support user and kernel mode
- But we cannot unify the building of the driver and tests because of this difference in toolchain
The goal (which may be impossible) would be to invoke our CMake for the driver as normal and have it also build the tests in the same command. This provides the benefit of ease-of-use for devs, unified compile commands, smooth integration of targets and dependencies.
Setup
The general structure of the project is:
ROOT
│ CMakeLists.txt
├───cmake
│ toolchain.cmake
├───src
│ lib.cpp
│ lib.h
└───tests
CMakeLists.txt
test.cpp
Everything is fairly standard. Our toolchain sets up the different compilers, flags, and such, with a switch KERNEL_MODE that determines if links to the KM or UM libraries should be used. The root CMake builds a KM library (part of the overall KMD). The tests CMake builds a UM library and the test EXE.
Of course, this last part is tricky since (as I’ve read on these forums) CMake only supports one toolchain per invocation. As such, I figure I will likely need to invoke the tests as a separate CMake call.
I will attach a sample project for this later.
Goals
- One build command to rule them all: Devs are already avoidant of tests so we need to make these as easy to use, modify, and build as possible.
- This means, ONE command to build the library and tests (that is
cmake -S ROOT). - No need to change LSP configuration to get intellisense on test files.
- This means, ONE command to build the library and tests (that is
- Reduce build times: The primary reason for upgrading to CMake was to reduce build times. Our main library greatly benefitted from this, but our unit tests still run make taking orders of magnitude longer to build then the library itself.
- This means we’d like to reduce unnecessary configuration steps and reduce recompilation.
- Clean and maintainable CMake: We should be able to continually add test and library files with ease and not need to add their data to multiple files.
- This means that we don’t want to introduce intertwining build logic where possible and avoid duplicating data such as source files or headers between CMake scripts.
Perhaps this is all a tall order. I’m an idealist though. Let’s get onto my approach for this problem:
Attempt 1: Flag Mangling
In my first approach, I simply added the test directory as a add_subdirectory and manually edited the flags to change the project to user mode.
# In `tests/CMakeLists.txt`
# We must override the kernel flags and includes for all of the testing targets.
# Due to CMake limiting us to one toolchain, we must be "dirty" and modify `CMAKE_` variables directly.
foreach(LANGUAGE C CXX)
string(REPLACE "-kernel" "" CMAKE_${LANGUAGE}_FLAGS "${CMAKE_${LANGUAGE}_FLAGS}")
string(REPLACE "${WDK_INCLUDE}/km/crt" "" CMAKE_${LANGUAGE}_STANDARD_INCLUDE_DIRECTORIES "${CMAKE_${LANGUAGE}_STANDARD_INCLUDE_DIRECTORIES}")
endforeach()
# We will also need to remove the omission of all default libraries when linking.
string(REPLACE "-NODEFAULTLIB" "" CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS}")
# We manually specify the user mode libraries directories.
target_link_directories(pp_utf PRIVATE
"${SDK_LIB}/um/${BUILD_ARCH}"
"${SDK_LIB}/ucrt/${BUILD_ARCH}"
"${VC_ROOT}/lib/${BUILD_ARCH}"
)
target_link_libraries(tests
kernel32.lib
user32.lib
gdi32.lib
comdlg32.lib
shell32.lib
bufferoverflowu.lib
)
Pros
This is (oddly) one of the simpler methods I tried. The root CMake has the least fiddling needed to achieve it since we simply add the tests as a subdirectory. In the tests CMake, once we set this up, we are good to go and can build the rest of the tests like usual. When doing clean builds, all objects we nicely removed.
We have access to all of the targets and definitions from the root CMake. This makes dependency management easier. As well, we can actually link our tests to the KM library (assuming we omit any calls to KM functions) and avoid the need to recompile our code.
Lastly, this gives me a unified compile_commands.json so that our LSP will work without modification for both the main library and tests sources. Previously this has been an issue that we had no LSP help when modifying tests (very cumbersome). Simply pointing our LSP to a “test compile commands” is infeasible because of the size of our library (Clangd will brick our machines trying to re-index the entire project, often taking 30+ minutes).
Cons
I needn’t say why this is not a great solution; It is incredibly fragile.
If something about the kernel mode configuration changes, our test build breaks (even if it’s switching /kernel to -kernel). As well, since we mangle the flags, we may break any kernel mode building that happens after our test’s configuration (we are one library of the driver with no control over when our library is configured).
Attempt 2: External Project
Here, I replaced the add_subdirectory with a call to ExternalProject_Add.
# In `ROOT/CMakeLists.txt`
include(ExternalProject)
ExternalProject_Add(mre_test
PREFIX ${TEST_BUILD_DIR}
SOURCE_DIR ${TEST_DIR}
CMAKE_GENERATOR ${CMAKE_GENERATOR}
CMAKE_ARGS
-DMRE_LIB_DIR=$<TARGET_FILE_DIR:mre_lib>
DOWNLOAD_COMMAND ""
INSTALL_COMMAND ""
DEPENDS mre_lib
BUILD_ALWAYS ON
)
Pros
This is also a relatively simple method, allowing for the test CMake to be largely standalone with no fancy logic in there.
This correctly tracks and rebuilds when we change either the test’s CMake file or source code.
This might be the canonical way of achieving my goal given the prevalence of the Super Build paradigm for cross compilation (which is essentially what I am doing).
Cons
It does not “feel” nice to have the tests as an external project. We have a separate invocation for configuring the tests and this can only happen at build time for the library. I really wish I could configure the tests at the configure time for the library (a sort of “IS_LOCAL” switch to the external project).
We do not have any access within the tests to the targets defined within the libraries CMake. This prevents easy reuse of, say, an interface library containing include paths for the library. As illustrated in the code snippet above, I have found a workaround to avoid recompilation of the library by passing the directory for the library as a definition. In the tests CMake, I would use this as:
target_link_directories(mre_test PUBLIC ${MRE_LIB_DIR})
target_link_libraries(mre_test PUBLIC mre_lib)
Though I unfortunately had to also manually add some of mre_lib’s link flags to the mre_test target (which is again, fragile). The same is true of the mre_lib header files.
Lastly, even if I pass-through the CMAKE_EXPORT_COMPILE_COMMANDS option, there is no information to compile test files within the compile commands file leading to cumbersome coding. I’m left with two compile commands files that I must switch between causing a large loss of productivity.
Attempt 3: Custom Commands
Here, I used the add_custom_command and add_custom_target to try and build the unit tests.
# In `ROOT/CMakeLists.txt`
add_custom_command(
OUTPUT
${TEST_EXE}
COMMAND
${CMAKE_COMMAND}
-G ${CMAKE_GENERATOR}
-B ${TEST_BUILD_DIR}
-S ${TEST_DIR}
COMMAND
${CMAKE_COMMAND}
--build ${TEST_BUILD_DIR}
VERBATIM
USES_TERMINAL
COMMENT "-- BUILDING UNIT TESTS --"
)
add_custom_target(test_exe
ALL
DEPENDS ${TEST_EXE} mre_lib
)
Pros
Almost nothing. This essentially only works on a clean build. This can link the libraries the same as we did in the external project.
Cons
Does not track changes to the test’s CMake or source files. Cannot produce a unified compile commands. Test configuration only happens at build time for the library.
Attempt 3.5: A Better Custom Command
The above is not really a contender at all. In order to make it any bit usable, we need it to track changes to the test files.
add_custom_command(
OUTPUT
${TEST_BUILD_DIR}/CMakeCache.txt
COMMAND
${CMAKE_COMMAND}
-G ${CMAKE_GENERATOR}
-B ${TEST_BUILD_DIR}
-S ${TEST_DIR}
--fresh
DEPENDS
${TEST_DIR}/CMakeLists.txt
VERBATIM
USES_TERMINAL
COMMENT "-- GENERATING UNIT TESTS BUILD --"
)
add_custom_target(test_exe_gen
ALL
DEPENDS ${TEST_BUILD_DIR}/CMakeCache.txt mre_lib
)
add_custom_command(
OUTPUT
${TEST_EXE}
COMMAND
${CMAKE_COMMAND}
--build ${TEST_BUILD_DIR}
DEPENDS
test_exe_gen
${TEST_BUILD_DIR}/CMakeCache.txt
${TEST_DIR}/test.cpp
VERBATIM
USES_TERMINAL
COMMENT "-- BUILDING UNIT TESTS --"
)
add_custom_target(test_exe
ALL
DEPENDS ${TEST_EXE}
)
Pros
This will track the changes to the CMake and source files for tests correctly (given that all source files are listed). The --fresh is needed to ensure the cache file is up-to-date (which means it’s largely a waste).
This has a nice feel to it. It splits the configuration and building steps cleanly apart. But that’s not worth much.
Cons
We have all the same cons of the external project. Additionally, it is intractable to reference every source file for the tests in any sizable project and would simply require large duplication of code. There is some possibility to have a TEST_SOURCES variable setup in an .cmake file and include it in the root and test CMake, but we’d still be tracking this list of dependencies twice.
An Aside For Split Compile Commands Files
I had experimented with writing a small function to merge the disjoint root and test compile_commands.json using jq. This was made difficult since the file itself is not created until the generation phase and so I cannot directly call a custom command on the files (they don’t exist at configuration time). Something like this in native CMake would be great as I know there would be nuance to the problem (such as my case where we would have the main library files repeated, once for KM and once for UM).
The Easy Out
Make 'em separate!
Sure, I could and I can’t deny that. But my point here is to illustrate a use case and some reasoning why I wouldn’t want to take that route. I think there are benefits, practical and aesthetic, to having a unified build. Otherwise, I wouldn’t have gone through all of this effort to create this post and do this background research.
Conclusion
After listing these options, it seems obvious that the external project method is the best and perhaps canonical answer. However, it is still unsatisfactory given that we are ‘mis-using’ it for an internal project. The fragmented compile commands, clunky use of targets from the parent project, and disjoint configuration steps stick out to me like a sore thumb.
I would like to hear the community and maintainer’s take on my situation and what could be done to improve it. What have I missed? Am I running a fools errand? How has anyone resolved a similar situation?
I hope you can appreciate the effort I am putting into this issue and I appreciate anyone who can give me a lead on this.
Future Work
Considering that CMake has this intrinsic “single-toolchain” setup, I wonder what the maintainers and community would think of built in support for a feature such as above (an InternalProject_Add if you will). One where we may share information on targets, configuration time, and compile commands as such. Even just an implementation of one piece of this system would make the above approaches better. I think simply being able to support multiple toolchains sounds like the “right option.”
If anyone knows what such a programming effort may entail, feel free to let me know. I have never looked into CMake’s source code and so I don’t have such a gauge. However, I am willing to give my time and effort for such a task if it is deemed acceptable and plausible.
Again, thank you for your time and effort reading my post.
Example Project
Here is a reference to a sample project in which you can play with the different approaches to this problem. Use the -DAPP_X=ON and -DBUILD_TESTS=ON flag at configuration to try them out. The toolchain is omitted, as it is specific to my work environment.