Project structure for unit testing and coverage

Usage
1

I’ve been reading a book by one of the CMake guys called “Professional Cmake.” It’s an amazing book and has been a daily tool for me lately.

However, one of the gray areas for me is not only how to structure a large project in terms of CMake best practices but, most importantly, how to set it up for unit testing from the start. The author does a great job explaining it, but I just can’t connect the dots for some reason.

I have many questions, but to narrow the scope of this post, I’ve consolidated them down to just two related queries. If I am lucky enough to get help on those topics, then the rest of my questions will fall into place (in theory).

I’ve created an example directory structure to aid in my questions. In reality, my structure is a bit different and contains many files, which would cloud the actual point.

With that said, in this example project, I can build and run just fine. I am cross-compiling between ARM and Windows, and have leveraged CMakePresets to do just that. Here’s a brief overview of the structure, using CMake 3.27, running on Windows.

Example_Project
|   CMakeLists.txt
|   CMakePresets.json
|   main.cpp
|   
+---app_a
|   |   CMakeLists.txt
|   |   
|   +---inc
|   |       app_a.h
|   |       
|   \---src
|           app_a.cpp
|           
+---app_b
|   |   CMakeLists.txt
|   |   
|   +---inc
|   |       app_b.h
|   |       
|   \---src
|           app_b.cpp
|           
+---app_c
|   |   CMakeLists.txt
|   |   
|   +---inc
|   |       app_c.h
|   |       
|   \---src
|           app_c.cpp
|           
+---drivers
|   +---adc
|   |   |   CMakeLists.txt
|   |   |   
|   |   +---inc
|   |   |       adc.h
|   |   |       
|   |   \---src
|   |           adc.cpp
|   |           
|   +---gpio
|   |   |   CMakeLists.txt
|   |   |   
|   |   +---inc
|   |   |       gpio.h
|   |   |       
|   |   \---src
|   |           gpio.cpp
|   |           
|   +---i2c
|   |   |   CMakeLists.txt
|   |   |   
|   |   +---inc
|   |   |       i2c.h
|   |   |       
|   |   \---src
|   |           i2c.cpp
|   |           
|   \---uart
|       |   CMakeLists.txt
|       |   
|       +---inc
|       |       uart.h
|       |       
|       \---src
|               uart.cpp
|               
\---tests
    |   CMakeLists.txt
    |   
    +---adc
    |       adc_test.cpp
    |       CMakeLists.txt
    |       
    +---app_a
    |       app_a_test.cpp
    |       CMakeLists.txt
    |       
    +---app_b
    |       app_b_test.cpp
    |       CMakeLists.txt
    |       
    +---app_c
    |       app_c.cpp
    |       CMakeLists.txt
    |       
    +---gpio
    |       CMakeLists.txt
    |       gpio_test.cpp
    |       
    +---i2c
    |       CMakeLists.txt
    |       i2c_test.cpp
    |       
    \---uart
            CMakeLists.txt
            uart_test.cpp

Before I write any more code, I want to figure out the unit testing and code coverage mechanisms with CMake. In this example project, I have integrated with GoogleTest, GoogleMock, and Gcov. I can run tests, and see the coverage report as HTML output.

Now for the CMake questions:



Question 1
I am seriously questioning my directory structure. In the book I referenced above, it mentions.

Instead of running ctest from the top of the build tree, it can be run from subdirectories below it. Only those tests defined from that directory’s associated source directory and below will be known to ctest. To be able to take full advantage of this, tests should not all be collected together in one place and defined with no directory structure. It may be useful to keep tests close to the source code they are testing so that the natural directory structure of the source code can be re-used to also give structure to the tests. If the source code is ever moved around, this approach also makes it easier to move the associated tests with it.

With that said, I wonder if my root-level “tests” directory is violating this suggestion. Maybe the better approach from a CMake point of view would be to have a “test” directory for each module. For example, the ADC driver, GPIO, UART, APP_(x) would all have their own separate test directory within the module itself, i.e next to the source.

My only pause on that is, if a new developer comes into the project and opens up the root of the project, there’s nothing that indicates where each test lives, as they are kind of tucked away out of sight, and might be neglected, as opposed to front-and-center in the root directory. I’ll have to think about that one.



Question 2
In terms of running the tests, I can run individual tests and produce coverage, but I can’t seem to figure out how to run all tests, and generate coverage.

Based on the directory structure above, the relationship looks like this:

  • The root level CMakeLists will include the tests/CMakeLists.txt if and only if this is a windows build and unit testing is enabled.

  • If enabled, the tests/CMakeLists.txt will then set up for CTest/googleTest and add each test directory, which also have its own CMakeLists.txt

Root level CMakeLists.txt

if (WIN32)
    if (UNIT_TESTS_ENABLED)
        add_subdirectory(tests)
    endif ()
endif()

tests/CMakeLists.txt

if (WIN32 AND UNIT_TESTS_ENABLED)

    # Setup for testing
    include(CTest)
    enable_testing()
    add_subdirectory(${CMAKE_CURRENT_LIST_DIR}/external/googletest)
    set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
    include(GoogleTest)
    include_directories(${GTEST_INCLUDE_DIR})
    list(APPEND CMAKE_CTEST_ARGUMENTS "--rerun-failed --output-on-failure")
    set(CTEST_OUTPUT_ON_FAILURE 1)

    # Add each test directory
    add_subdirectory(${CMAKE_CURRENT_LIST_DIR}/i2c)
   # … Shortened for brevity
endif ()

tests/i2c/CMakeLists.txt

set(LIB_UT_NAME i2c_tests)

# Pull in gtest module
include(GoogleTest)

# Link necessary libraries
target_link_libraries(${LIB_UT_NAME } gtest gtest_main gmock i2c})
target_include_directories(${LIB_UT_NAME } PRIVATE ${GTEST_INCLUDE_DIRS})
target_link_libraries(${LIB_UT_NAME } PRIVATE ${GTEST_LIBRARIES})

# Use gtest_discover_tests to handle the individual test cases
gtest_discover_tests(${LIB_UT_NAME })

In this scheme, when built, each unit test will have its own directory under the root level build directory, which contains its executable. For example build/tests/i2c/CmakeFiles/i2c_test.exe

I can then independently run that .exe from the command line. Or, I can navigate to the directory and run ctest. The issue I am having, again probably related to my project structure, is that I want the developer to be able to run their specific tests and see coverage for only the test that has run.

However, I also want the developer to run all tests from the root of the project, which would generate coverage for every single test. The issue is, I feel like I am reinventing the wheel. I’m sure this is a common use case, and right now, my solution is a hack. Basically, in each of the tests, I have something like this, which runs the tests, and then generates the coverage, as a post-build step.

tests/i2c/CMakeLists.txt

# Dump coverage after tests
add_custom_command(
        TARGET ${LIB_UT_NAME} POST_BUILD
        COMMAND ctest .
        COMMAND Echo "CTest complete, generating code coverage"
        COMMAND ${CMAKE_COMMAND} -D BIN_DIR=${CMAKE_CURRENT_BINARY_DIR} -P ${CMAKE_SOURCE_DIR}/run_coverage.cmake
        WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
        COMMAND Echo "Code coverage complete and has been written to coverage.html"

In this case, the developer would run:
cmake –build –preset foo –target i2c_tests

This will build the module, test it, and then run code coverage. However, I am calling ctest from within a CMakeLists.txt file, and this just feels wrong. I do have a “testPreset” setup in my CMakePresets.json, but I can’t figure out how to use it in conjunction with coverage.

In the end, my goal is to be able to let the developers build either a single test, or all tests, and generate coverage in both cases.

Sorry for the long post, but I feel like I’ve gone down the wrong path, so I wanted to reach out and see what others have done, or maybe there’s a good example repo out there to reference in terms of CMake/CTest/GoogleTest and coverage integration.

Thanks!

2 Likes
2

Glad you’re finding the book useful. I’ll split up my reply into separate posts to make them easier to digest.

Your suspicion is correct. I would recommend keeping unit tests close to the code they are testing. In your scenario, that would mean putting unit tests with each module. For example, part of your directory structure would then look something like this:

Example_Project
|   CMakeLists.txt
|   CMakePresets.json
|   main.cpp
:
+---drivers
|   +---adc
|   |   |   CMakeLists.txt
|   |   |
|   |   +---inc
|   |   |       adc.h
|   |   |
|   |   +---src
|   |   |       adc.cpp
|   |   |
|   |   \---tests
:   :           CMakeLists.txt
:   :           test_adc.cpp

Note that the above recommendation is for unit tests. For integration tests that span across modules, putting those under a top level tests directory like you’ve done would still be appropriate.

3

Keep an eye out for the next edition, which should contain new material about code coverage. :wink:

Normally, a developer would run ctest from the top build directory. By default, it will run all tests, but you can use various forms of filtering to only run a subset or individual tests. Look at the ctest options like -R and -E, which allow you to use regular expressions on the test names. There’s also -L and -LE which do similar based on test labels, or -I if you want to pick out tests by test number (generally much less convenient). See the “Test Grouping And Selection” section of the “Testing” chapter of the book which covers these. The book mentions that you can also run ctest from a subdirectory. While that does generally work, getting familiar with the test selection options and running things from the top of the build directory is probably the better longer term strategy. It’s more conventional, and is directly supported (I’m not sure if running from a subdirectory is officially supported or just happens to work - there are some scenarios where it won’t, or at least won’t work as good).

For the part about collecting coverage results, this is where things get a little less clear cut. It looks like you’ve already captured the steps for processing coverage results in a run_coverage.cmake script, which should simplify discussion here. Rather than trying to put both running ctest and processing the results in the one custom command, I’d leave out the running ctest part and create a build rule just for processing the coverage data (running your run_coverage.cmake script). Now the developer can choose how they want to run ctest, whether for all tests or just a subset, and then they can run the same coverage script afterward to process whatever results have been generated.

The above suggestion works very well with CMake’s workflow presets. You can define workflows which do the build, run tests, then build the custom target that executes your run_coverage.cmake script to collect the results. I’ve done this with some of my consulting clients, and it is quite easy to use. Admittedly those cases were more aimed at continuous integration pipelines, but developers could adapt it to their needs as required.

Rather than make assumptions about the finer details of what you’re wanting to achieve, I’ll hold off commenting further on the above in case I’ve misunderstood what you’re aiming for.

1 Like
4

A few comments about this particular block of code:

  • Unless you’re running a CDash Dashboard script, there’s little benefit to doing include(CTest). That adds a bunch of noise for things intended for nightly builds, but it just tends to get in the way for developers. I don’t typically include that module in any of my projects these days.
  • The call to enable_testing() needs to be in the top level CMakeLists.txt file. I’d put it right after the first project() call (you can still put it inside an if(WIN32) test if that’s the only platform tests should be run on).
  • You would probably also move the add_subdirectory(.../googletest) through to include(GoogleTest) lines up to the top level CMakeLists.txt as well, since you will want each module to have access to gtest once you move your unit tests into each module instead of under a top level tests area.
  • Those two lines look reversed to me.
  • Remove this. Link your test targets to GTest::gtest or GTest::gtest_main instead. The header search paths will then be added transitively for you.
  • The project shouldn’t set these. These things are meant to be developer controls. You can put them in test presets instead, which still allows the developer to override them if they want to.

In your next code block, you have this:

The last two lines are unnecessary. Linking to the right gtest libraries will automatically add the header search paths and necessarily libraries to be linked. The following should be all you need in this case:

target_link_libraries(${LIB_UT_NAME} GTest::gtest_main GTest::gmock i2c)

Your code also seems to have stray } at the end of the equivalent line, and also an extra space in ${LIB_UT_NAME }, but I’m assuming that’s just a result of the way you’ve cut this down from your real project.

2 Likes