Testing a lib with various build configurations

Usage
1

Hello,

I’m currently trying to set-up a small embedded-C project using CMake. Part of the code is conditionally built using preprocessor guards. The CMake project uses an option() to enable/disable this. Using generator expressions, the define is passed from cmake to the compiler. This basically looks like this:

option(OPTION_A "Option A description")
add_library(my_target STATIC my_target.c)
target_compile_definitions(my_target PUBLIC $<$<BOOL:${OPTION_A}>:OPTION_A>)

Now I’m trying to use ctest to run various tests. Of-course I would like to test my code both when the option is enabled or disabled. For the moment I use add_test, but I can only test one version of the code at once.

add_executable(test_my_target test_my_target.c)
target_link_libraries(test_my_target PRIVATE my_target)
add_test(NAME test_my_target COMMAND test_my_target)

I would like to be able to define two tests i(one exploiting the option, the other not), that I could run at once. By that I mean that I don’t want to contantly reconfigure the project and rebuild then re-run the tests (I agree that at some point I’ll need to,build the lib and it’s dependency twice).

I started having a look at CTest “Build and Test Mode”, but for now I wasn’t able to achieve what I wanted. If that is a good direction to take, I’ll re-read the doc and persevere.

My questions are the following:

  • Is what I want to achieve a good idea ?
  • Is this achievable in a simple manner ?
  • Is CTest “Build and Test Mode” a way to achieve this ?
2

Since it comes in through a -D flag, you can have two tests, one does:

#ifdef OPTION_A
#undef OPTION_A
#endif

#include "header/from/my_target"

// tests without option set

#ifndef OPTION_A
#define OPTION_A
#endif

#include "header/from/my_target"

// tests with option set

However, this assumes that the option is purely an interface thing (i.e., my_target’s compilation itself doesn’t care about OPTION_A). Using INTERFACE instead of PUBLIC would help show this.

However, if the code depends on OPTION_A, you’ll need to compile my_target twice: once with the option set and once without.

3

Thanks @ben.boeckel for the answer. Unfortunately the compilation cares about OPTION_A, this is not just a header thing.

I tried a few things and I’m currently in the following state:

$ tree                                                                                                                                                                                                           
.
├── src
│   ├── CMakeLists.txt
│   ├── my_lib.c
│   └── my_lib.h
├── tst
│   ├── CMakeLists.txt
│   └── my_lib_tst.c
└── tst.sh

# src/CMakeLists.txt
cmake_minimum_required(VERSION 3.28)
project(my_lib)

option(OPTION_A "Option A description")

add_library(my_lib STATIC my_lib.c)
target_include_directories(my_lib PUBLIC .)
target_compile_definitions(my_lib PUBLIC $<$<BOOL:${OPTION_A}>:OPTION_A>)

// src/my_lib.c
#include "my_lib.h"
int func(void)
{
#ifdef OPTION_A
  return 0;
#else
  return 1;
#endif
}

// src/my_lib.h
#ifndef MY_LIB_H
#define MY_LIB_H

int func(void);

#endif // MY_LIB_H

# tst/CMakeLists.txt                                                                                                                                                                                              
cmake_minimum_required(VERSION 3.28)
project(my_lib_tst)

add_subdirectory(../src ${CMAKE_CURRENT_BINARY_DIR}/src)

add_executable(my_lib_tst my_lib_tst.c)
target_link_libraries(my_lib_tst PRIVATE my_lib)

Currently the test files can use public defines from the lib.

// tst/my_lib_tst.c
#include "my_lib.h"

int main (void)
{
#ifdef OPTION_A
  return (func() == 0) ? 0 : -1;
#else
  return (func() == 1) ? 0 : -1;
#endif
}

And I’m driving the tests using a shell script and ctest in “Build and Test Mode”

ctest --build-and-test tst/ tst/build_on --build-generator Ninja \
  --build-project my_lib_tst \
  --build-options -DOPTION_A=ON \
  --test-command my_lib_tst

ctest --build-and-test tst/ tst/build_off --build-generator Ninja \
  --build-project my_lib_tst \
  --build-options -DOPTION_A=OFF \
  --test-command my_lib_tst

If this works as expected, the output is not easy to parse for a human, and I would like to drive the test execution using ctest.

I might be missing something but I’m unable to do that for now. Here is what I’ve tried so far:

cmake_minimum_required(VERSION 3.28)
project(test)

add_test(tst1
  ${CMAKE_CTEST_COMMAND}
  --build-and-test tst/ tst/build_on
  --build-generator ${CMAKE_GENERATOR}
  --build-makeprogram ${CMAKE_MAKE_PROGRAM}
  --build-project my_lib_tst
  --build-options -DOPTION_A=ON
  --test-command my_lib_tst)

add_test(tst2
  ${CMAKE_CTEST_COMMAND}
  --build-and-test tst/ tst/build_off
  --build-generator ${CMAKE_GENERATOR}
  --build-makeprogram ${CMAKE_MAKE_PROGRAM}
  --build-project my_lib_tst
  --build-options -DOPTION_A=OFF
  --test-command my_lib_tst)

This last bit is taken from Using CTest to Drive Complex Tests

4

VTK runs things like this here:

https://gitlab.kitware.com/vtk/vtk/-/blob/master/Examples/CMakeLists.txt

5

Thanks once again for the answer.

This kind of help me, as I realized that having some relative path like I had might not be the best idea.

The other missing part was that I also needed to include CTest in the CMakeLists.txt.

# CMakeLists.txt
cmake_minimum_required(VERSION 3.28)
project(test)

include(CTest)

add_test(NAME tst1 COMMAND
  ${CMAKE_CTEST_COMMAND}
  --build-and-test ${CMAKE_CURRENT_SOURCE_DIR}/tst/ 
  ${CMAKE_CURRENT_BINARY_DIR}/tst/build_on
  --build-generator ${CMAKE_GENERATOR}
  --build-makeprogram ${CMAKE_MAKE_PROGRAM}
  --build-project my_lib_tst
  --build-options
      -DOPTION_A=ON
  --test-command my_lib_tst)

add_test(NAME tst2 COMMAND
  ${CMAKE_CTEST_COMMAND}
  --build-and-test ${CMAKE_CURRENT_SOURCE_DIR}/tst/
  ${CMAKE_CURRENT_BINARY_DIR}/tst/build_off
  --build-generator ${CMAKE_GENERATOR}
  --build-makeprogram ${CMAKE_MAKE_PROGRAM}
  --build-project my_lib_tst
  --build-options
      -DOPTION_A=OFF
  --test-command my_lib_tst)

This is for now giving me the result I want. Still need to dig a bit more I guess, but as far as the original issue goes, this seems to be resolved.

6

You probably don’t actually want or need to include(CTest). That’s more aimed at people using CDash. You most likely want to call enable_testing() instead, which won’t create all the additional CDash-related targets that include(CTest) does.

1 Like
7

Thanks for the feedback,

I’m now wondering, in the case the CMakeLists is not that simple and includes other not-test-related things, should I use the following pattern, so I can enable or not the build of tests from the command line ? (Being able to choose if tests are built or not was the reason I didn’t use enable_testing() in the first place)

# CMakeLists.txt
cmake_minimum_required(VERSION 3.28)
project(my_project)

add_executable(...)
add_library(...)

if (BUILD_TESTING)
   enable_testing()
   add_test(...)
endif()
8

This is a common misunderstanding. The enable_testing() command does not determine whether your tests are built. It only determines whether CMake writes out the necessary files for ctest. If you want to avoid building tests based on some cache option, you have to implement that yourself. As it happens, the CTest module does help with part of doing that (it defines the BUILD_TESTING option, but it’s still up to the project to use it), but CTest brings a whole lot of other things (i.e. targets associated with CDash) which most projects don’t actually want.

Frankly, I wish we didn’t need to call enable_testing() at all. In my opinion, every project should just call it straight after its top level call to project(). I’m yet to see a case where this had any measurable impact on generation time, or where it led to any problems at all. The advantage is that now your project always supports ctest, if you define any tests. Here’s the pattern I typically recommend:

cmake_minimum_required(VERSION 3.28)   # Must be at least 3.21 for PROJECT_IS_TOP_LEVEL
project(my_project)
enable_testing()   # No harm even you ultimately don't add any tests

add_executable(...)
add_library(...)

if(PROJECT_IS_TOP_LEVEL)
   add_test(...)
endif()

Some variations on the above are also possible. If you want the flexibility to turn tests on or off when your project is the top level project (which it sounds like you do), then you can expand the if() condition to account for that. Some people want to allow the tests to be added to the build even if the project is not the top level. Occasionally that’s useful, but most of the time it isn’t. For the case where you want that degree of flexibility, here’s how I’d implement that:

cmake_minimum_required(VERSION 3.28)
project(my_project)
enable_testing()

add_executable(...)
add_library(...)

option(MYPROJECT_ENABLE_TESTS "Enable tests for my_project" ${PROJECT_IS_TOP_LEVEL})
if(MYPROJECT_ENABLE_TESTS)
   add_test(...)
endif()
2 Likes