CTest Dynamic Analysis

Hi everyone,

I currently explore the possibilities of cmake scripting to run sanitizers and tests. But it seems to be not working.

# This file is supposed to run in ctest script mode:
# ctest -S <path-to-this-file>/CTestScript.cmake
# You can set some command line variables to change the behaviour of this script:
# -DCTEST_MODEL:STRING=Experimental|Nightly|Continuous
# -DCTEST_CMAKE_GENERATOR:STRING=Ninja|Unix Makefiles|...)

# Memory checker (MemoryCheckCommand) not set, or cannot find the specified program. -> Doesn't stop the CI


  set(CTEST_CMAKE_GENERATOR "Unix Makefiles")


  set(CTEST_MEMORYCHECK_TYPE "AddressSanitizer")
    "-DCMAKE_CXX_FLAGS_INIT=-fsanitize=address -fno-omit-frame-pointer"
    "-DCMAKE_EXE_LINKER_FLAGS_INIT=-fsanitize=address -fno-omit-frame-pointer"
  # set(CTEST_MEMORYCHECK_SANITIZER_OPTIONS "verbosity=1:exitcode=-1:check_initialization_order=true:detect_stack_use_after_return=true:strict_init_order=true:detect_invalid_pointer_pairs=10:strict_string_checks=true")
  set(CTEST_MEMORYCHECK_TYPE "MemorySanitizer")
    "-DCMAKE_CXX_FLAGS_INIT=-fsanitize=memory -fno-omit-frame-pointer"
    "-DCMAKE_EXE_LINKER_FLAGS_INIT=-fsanitize=memory -fno-omit-frame-pointer"
  # set(CTEST_MEMORYCHECK_SANITIZER_OPTIONS "verbosity=1:exitcode=-1")
  set(CTEST_MEMORYCHECK_TYPE "LeakSanitizer")
    "-DCMAKE_CXX_FLAGS_INIT=-fsanitize=leak -fno-omit-frame-pointer"
    "-DCMAKE_EXE_LINKER_FLAGS_INIT=-fsanitize=leak -fno-omit-frame-pointer"
  # set(CTEST_MEMORYCHECK_SANITIZER_OPTIONS "verbosity=1:exitcode=-1")
  set(CTEST_MEMORYCHECK_TYPE "ThreadSanitizer")
    "-DCMAKE_CXX_FLAGS_INIT=-fsanitize=thread -fno-omit-frame-pointer"
    "-DCMAKE_EXE_LINKER_FLAGS_INIT=-fsanitize=thread -fno-omit-frame-pointer"
  # set(CTEST_MEMORYCHECK_SANITIZER_OPTIONS "verbosity=1:exitcode=-1")
  set(CTEST_MEMORYCHECK_TYPE "UndefinedBehaviorSanitizer")
    "-DCMAKE_CXX_FLAGS_INIT=-fsanitize=undefined -fno-omit-frame-pointer"
    "-DCMAKE_EXE_LINKER_FLAGS_INIT=-fsanitize=undefined -fno-omit-frame-pointer"
  # set(CTEST_MEMORYCHECK_SANITIZER_OPTIONS "verbosity=1:exitcode=-1:print_stacktrace=1")
  # find_program(CTEST_MEMORYCHECK_COMMAND NAMES valgrind)
  # set(CTEST_MEMORYCHECK_COMMAND_OPTIONS "--errors-for-leak-kinds=all --show-leak-kinds=all --leak-check=full --error-exitcode=1")

  set(CTEST_MODEL "Continuous")

# Dashboard actions to execute, always clearing the build directory first
ctest_configure(OPTIONS "${configureOpts}")

And I then run the script with:
and again with the other sanitizers.

It also uploads to the CDash, but it says everything is alright and I built this in the main.cpp:

    // test memory leak
    double* my_array = new double[1000];
    std::cout << my_array;

When I run manually with LSAN it actually shows the memory leak.

I do get the following error:

Cannot find memory tester output file: /home/leon/Dokumente/Projects/beans/build/Testing /Temporary/MemoryChecker.1.log.*

So what am I missing? What sources / targets are actually checked? And why do I have to specify the flags if I already set set(CTEST_MEMORYCHECK_TYPE “LeakSanitizer”).

I would also be greatfull for helpful resources. I found a few words and example in “Professional cmake” and other sources, but it wasn’t so detailed.

In the end, I want to actually not use CDash at all, but rather use it with gitlab pipeline. But I figured, starting with CDash might be easier.

Not sure if it is relevant, but see if this helps: https://gitlab.kitware.com/cmake/cmake/-/issues/19482

And possibly this too: https://gitlab.kitware.com/cmake/cmake/-/issues/16448

@craig.scott At least I did find the cmake script https://gitlab.kitware.com/cmake/dashboard-scripts/-/blob/master/cmake_common.cmake powering dashboard. I searched for that but didn’t expect it to be in a different repo. That might be a help.
But it seems like cmake isn’t using Dynamic analysis.

Apart from that, these threads seem not to be related to my problem unfortunately. I also tried the script from your book, which uses an AddressSanitizer. It didn’t work either (I only replaced it by Leak Sanitizer and the CXX Flag with leak).

Btw: Why does cmake use different variables like CTEST_BUILD_CONFIGURATION instead of CTEST_CONFIGURATION_TYPE? I can’t find documentation for the first one (same with other stuff)

I tried your CTestScript.cmake on an Ubuntu 18 VM, but with the last ctest_submit() line commented out since I wasn’t interested in the submission part. I created an empty CTestConfig.cmake file and a sample project which created an executable with your code sample (see below). I invoked it with the same command line you did and it all worked for me. The leak was successfully detected. I suggest you append -VV to the ctest command line and see if the extra output gives any clues as to why your case is behaving differently.

You can also inspect the CMakeCache.txt in the build directory created by running the dashboard script. You should see the relevant -fsanitize option in the relevant CMake variables.


cmake_minimum_required(VERSION 3.18)


add_executable(leaker main.cpp)

add_test(NAME leaker COMMAND leaker)


#include <iostream>

int main()
    double* my_array = new double[1000];
    std::cout << my_array;

Command line:



* Extra verbosity turned on Reading Script: /home/craig/Projects/minimal/CTestScript.cmake SetCTestConfiguration:SourceDirectory:/home/craig/Projects/minimal SetCTestConfiguration:BuildDirectory:/home/craig/Projects/minimal/build Run dashboard with model Continuous Source directory: /home/craig/Projects/minimal Build directory: /home/craig/Projects/minimal/build Reading ctest configuration file: /home/craig/Projects/minimal/CTestConfig.cmake SetCTestConfigurationFromCMakeVariable:Site:CTEST_SITE SetCTestConfiguration:Site:ubuntu-18-lts SetCTestConfigurationFromCMakeVariable:BuildName:CTEST_BUILD_NAME SetCTestConfiguration:BuildName:Linux Site: ubuntu-18-lts Build name: Linux Use Continuous tag: 20200716-1139 SetCTestConfiguration:BuildDirectory:/home/craig/Projects/minimal/build SetCTestConfiguration:SourceDirectory:/home/craig/Projects/minimal SetCTestConfiguration:ConfigureCommand:"/snap/cmake/487/bin/cmake" "-DCMAKE_CXX_FLAGS_INIT=-fsanitize=leak -fno-omit-frame-pointer" "-DCMAKE_EXE_LINKER_FLAGS_INIT=-fsanitize=leak -fno-omit-frame-pointer" "-DCMAKE_BUILD_TYPE:STRING=LSAN" "-GUnix Makefiles" "/home/craig/Projects/minimal" Configure project Configure with command: "/snap/cmake/487/bin/cmake" "-DCMAKE_CXX_FLAGS_INIT=-fsanitize=leak -fno-omit-frame-pointer" "-DCMAKE_EXE_LINKER_FLAGS_INIT=-fsanitize=leak -fno-omit-frame-pointer" "-DCMAKE_BUILD_TYPE:STRING=LSAN" "-GUnix Makefiles" "/home/craig/Projects/minimal" Run command: "/snap/cmake/487/bin/cmake" "-DCMAKE_CXX_FLAGS_INIT=-fsanitize=leak -fno-omit-frame-pointer" "-DCMAKE_EXE_LINKER_FLAGS_INIT=-fsanitize=leak -fno-omit-frame-pointer" "-DCMAKE_BUILD_TYPE:STRING=LSAN" "-GUnix Makefiles" "/home/craig/Projects/minimal" -- The C compiler identification is GNU 9.3.0 -- The CXX compiler identification is GNU 9.3.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Configuring done -- Generating done -- Build files have been written to: /home/craig/Projects/minimal/build Command exited with the value: 0 SetCTestConfiguration:BuildDirectory:/home/craig/Projects/minimal/build SetCTestConfiguration:SourceDirectory:/home/craig/Projects/minimal SetMakeCommand:/snap/cmake/487/bin/cmake --build . --config "LSAN" -- -i SetCTestConfiguration:MakeCommand:/snap/cmake/487/bin/cmake --build . --config "LSAN" -- -i Build project MakeCommand:/snap/cmake/487/bin/cmake --build . --config "LSAN" -- -i Run command: "/snap/cmake/487/bin/cmake" "--build" "." "--config" "LSAN" "--" "-i" Scanning dependencies of target leaker [ 50%] Building CXX object CMakeFiles/leaker.dir/main.cpp.o [100%] Linking CXX executable leaker [100%] Built target leaker Command exited with the value: 0 MakeCommand:/snap/cmake/487/bin/cmake --build . --config "LSAN" -- -i 0 Compiler errors 0 Compiler warnings SetCTestConfiguration:BuildDirectory:/home/craig/Projects/minimal/build SetCTestConfiguration:SourceDirectory:/home/craig/Projects/minimal SetCTestConfigurationFromCMakeVariable:MemoryCheckType:CTEST_MEMORYCHECK_TYPE SetCTestConfiguration:MemoryCheckType:LeakSanitizer Memory check project /home/craig/Projects/minimal/build Constructing a list of tests Done constructing a list of tests Updating test list for fixtures Added 0 tests to meet fixture requirements Checking test dependency graph... Checking test dependency graph end test 1 Start 1: leaker Memory check command: /snap/cmake/487/bin/cmake "-E" "env" LSAN_OPTIONS=log_path='/home/craig/Projects/minimal/build/Testing/Temporary/MemoryChecker.1.log'

1: MemCheck command: /snap/cmake/487/bin/cmake “-E” “env” “LSAN_OPTIONS=log_path=’/home/craig/Projects/minimal/build/Testing/Temporary/MemoryChecker.1.log’” “/home/craig/Projects/minimal/build/leaker”
1: Test timeout computed to be: 600
1: 0x624000000000
1/1 MemCheck #1: leaker …***Failed 0.02 sec
1: process test output now: leaker leaker
PostProcessTest memcheck results for : leaker
Remove: /home/craig/Projects/minimal/build/Testing/Temporary/MemoryChecker.1.log.15690

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.02 sec

The following tests FAILED:
1 - leaker (Failed)
– Processing memory checking output:
1/1 MemCheck: #1: leaker … Defects: 1
MemCheck log files can be found here: (<#> corresponds to test number)
Memory checking results:
Direct leak - 1
Test project /home/craig/Projects/minimal/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph…
Checking test dependency graph end
test 1
Start 1: leaker

1: Test command: /home/craig/Projects/minimal/build/leaker
1: Test timeout computed to be: 600
1: 0x624000000000
1: =================================================================
1: ==15692==ERROR: LeakSanitizer: detected memory leaks
1: Direct leak of 8000 byte(s) in 1 object(s) allocated from:
1: #0 0x7f2c47422cbd in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/liblsan.so.0+0xfcbd)
1: #1 0x55e213f20906 in main (/home/craig/Projects/minimal/build/leaker+0x906)
1: #2 0x7f2c46c6fb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
1: SUMMARY: LeakSanitizer: 8000 byte(s) leaked in 1 allocation(s).
1/1 Test #1: leaker …***Failed 0.02 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.02 sec

The following tests FAILED:
1 - leaker (Failed)

First of all, thank you for your incredible effort you made!

It made me realize that I fundamentally didn’t understand how these sanizers / cmake scripts and tests play together. And perhaps I also didn’t understand tests in CMake that well.

So the problem seem to be that I have not declared a test (the right test). The main problem seems to be that I have no idea what should be a test and what shouldn’t.

For me this is a test (I’m not very experiences with tests I have to admit):

TEST_CASE( "Factorials are computed", "[factorial]" ) {
    REQUIRE( Factorial(1) == 1 );
    REQUIRE( Factorial(2) == 2 );
    REQUIRE( Factorial(3) == 6 );
    REQUIRE( Factorial(10) == 3628800 );

And I might have multiple of them in multiple files.

Now CMake seems to abstract from that you can have multiple tests, which consist of multiple files and multiple test cases. So that already confuses me, because I really don’t know how to structure it. Should I have multiple tests for one executable / library?

So currently in my projekt I only have one application and I added tests, so what I did is instead of having

add_executable(<MyApp> someSources)

I changed it to

add_library(<MyApp> STATIC sameSourcesAsAboveButWithoutMainCpp)
target_include_directories(<MyApp> PUBLIC .) #So I find the header files for my test, which is in a different test directory
add_executable(<MyApp>App Main.cpp)
target_link_libraries(<MyApp>App PRIVATE MyApp)

So I could add a test and link against my executable

add_executable(<MyApp>Test main.cpp) # this is a different main.cpp in a test folder with some TEST_CASE macros
target_link_libraries(<MyApp>Test PRIVATE <MyApp> Catch2::Catch2)

add_test(NAME test<MyApp> COMMAND <MyApp>Test)

So you see I already have naming problems. But was that approach correct?

I got slightly confused by this now

add_executable(leaker main.cpp)
add_test(NAME leaker COMMAND leaker)

So you create a test, which executes a normal target?

I can somehow imagine that this might be needed for the sanitizers, but it seems weird to me anyway. And also a little bit problematic:

  1. My Application is a bit python like, so if you just start it, it starts in a “run mode”, where you can execute multiple commands. So when I added a test for my Application it required user input to get out of that endless mode.
  2. If I just run my normal tests (as part of my CI with cmake --build buid -t tests) this will probably be executed as a test. But it can’t then actually fail, can it? … I could probably live with that, but it seems odd to have tests, which always pass

Would you mind, going more into detail, why I need this weird looking test and how to resolve my issue(s) with that approach (at least the first one -> actually I also didn’t get the error then, but I did if I just created a main as you did). Maybe also some general advice on how to structure tests if you find the test.

Thank you a lot!

By the way: Do you accept / want feedback for your book? It’s a great book overall, but while working with it, I was a few times in the situation that there was slightly to less information about a topic or an important detail of a certain topic was covered in a later chapter, where the book could profit from a reference to that. I would like to share some of my thoughts, perhaps in the forum here, if you would appreciate it.

The add_test() command defines a command to run for that test case. You can define test properties to control how ctest decides whether a test passed, failed or was/should be skipped. In some cases, you might have an executable that uses a testing framework and the executable contains multiple test cases. For that, you may want to have ctest invoke that executable multiple times with different arguments to execute each test individually. This is functionality that the GoogleTest module provides, for example. How you structure your tests and executables is up to you.

That approach seems okay to me. Not sure what naming problems you are referring to here. See next comment below though.

Yes. The NAME and COMMAND don’t have to be the same, that just made sense in this case. The COMMAND can also have arguments, it doesn’t have to be just a bare target name. It can be anything that can be executed at test time. When you have sanitisers enabled, it still runs the command but if that command is a target built by the project, it should exit with a non-zero exit code if there are problems (I think) and by default ctest would then see it as a failed test, even if the actual code completed successfully in terms of its logic and what it was trying to test.

If you want it to run in CI, expecting user input is going to be problematic. Perhaps you can modify how your application works so that it accepts a script and when it reaches the end of the script, the application ends. This is basically what languages like python and shell scripts do.

If the application ends with a non-zero exit code, that will be seen as a failure by default. As mentioned above, you can also set test properties to match the output for pass or fail regular expressions.

Feedback on the book is always welcome. This forum is for CMake itself though, my book is a private endeavour. A more appropriate place to provide feedback on the book would be the Crascit website:

Thanks a lot, the post already helped a lot!

Yeah I actually was more confused that you just link a normal target, by “normal” I mean here, one without any test cases. That didn’t make any sense to me, when I first saw that. But I understand that you might need this to just check if your application isn’t crashing and in particular you need it for the sanitizers.
I do have one question though, which is not really related to cmake thought, but perhaps you have some experience with anyway. Is it correct then that only code is tested by sanitizers, which is actually run? So say I have functionA and functionB and by default if I invoke my programm only functionA is called. Function B is only called if you pass some param. Then sanitizers won’t check functionB for memory leaks and other stuff? (unless also called with that special command line param)
So if I want my application to be checked completly I need to ensure that the applications runs in a way (or multiple times) that all code is called once?

Great! I will start writing down what could be improved or is unclear from my perspective, hopefully that will be a help for you.

Correct. The sanitisers can only check code paths that are actually executed. If you have poor code coverage in your tests, then enabling sanitisers will be less effective.

1 Like

That makes sense! So with that knowledge I will try to improve my setup and let’s see how that goes.