Good morning.
I’ve recently been working on the build system for a project of mine, and, desiring to keep the dependency list low, I have opted to write my conditional build logic in CMake. This has led me to write some independent scripts where their function is crucial to the build system. In support of testing these scripts, I have written a basic test generation framework.
However, I have come to a problem. This framework takes in valid CMake files as input and uses fake macros to determine how to set up the test groups and fixtures. The thing is that as I iterate on how to handle conditional macro calls and said calls being imported via include statements, I am beginning to suspect it would be easier to add this test generation functionality into the CMake binary itself.
How my framework is intended to work in practice is to run the unit tests prior to parsing the build targets. Should any of the tests fail, so does the build file generation. How I would implement this would be through some built-in
“run_cmake_script_tests(TEST_DESCRIPTOR_FILE)” function, where the argument points either to a test descriptor file or to a CMake file that includes all the test descriptor files.
Would this be something you would want in the CMake binary itself?
1 Like
That seems like quite a specific use case, so I’d be skeptical that it would be upstreamed
Really? Honestly, I thought that using CMake scripts to facilitate optional build logic resolution was fairly common. If so, it would make sense to be able to test these scripts.
Looking through your linked repo, I’m a bit confused about what exactly it’s doing… it’s Python code that generates CMake code capable of testing some other CMake code, and you’ve got CMake code to run the Python script and execute the generated CMake test code. I don’t really understand why you’re doing any of this, or what the value is…?
In another project, I have scripts containing custom CMake logic that direct the build based on certain parameters. This project supports multiple OSs who each have different very different ways of achieving the same functionality. Hence, the code must be modular with certain modules being OS-dependent. My scripts are used to handle certain preparation tasks (automating generating VS Code debug launch profile JSON, for instance) and for handling OS-dependent build preparation logic. My test repo is for generating tests to ensure these scripts continue to operate correctly.
As to why it’s CMake-nearly-all-the-way-down, it’s for fluidity of thought for the developer and ease of test definition. Tests are defined in real CMake code, making them easy to reason about. Unfortunately, CMake does not inherently define a way for setting up test fixtures. Hence, why the test code is generated via Python and then executed in its own process.
Think of it from the perspective of someone who is merely incorporating my project. I want to make it easy for them. Almost as easy as using GoogleTest for debugging C++ code. In doing so, I also make it easy for Future Me to continue to use this code, which is nice considering I intent to keep using CMake to generate build files for all future projects involving native code.
How exactly is your test framework testing your code, though? Reading through it, all I can find is that it’s checking that certain macros/functions exist.
In my projects I do have some configure-time test logic that simply gets executed as part of the configure step, I don’t really understand the need for generating a separate script to be executed via cmake -P. In general, I don’t really have high confidence in unit tests that are themselves auto-generated.
It does more than that. It uses those “fake-macros”, as I call them, to set up the test structures. Try it by running “cmake -P framework-tester.cmake”. This script is used to test the testing framework itself during development. After running, take a look at the generated code in “python/tests”. The fake-macros are used to define how to structure the calls to the real macros: the setup macro, the teardown macro, and the test macros.
It’s nowhere near perfect, and I am in the process of making improvements (adding basic CMake variable dereferencing for certain variables to ensure the generated scripts work correctly).