MemoryCheckCommand for CTest Memcheck with [Address/Thread/Memory]Sanitizer

Usage
1

Hi,

I’m trying to setup CTest memcheck for build with sanitizers. I have an issue with missing MemoryCheckCommand/CTEST_MEMORYCHECK_COMMAND setting:

Memory checker (MemoryCheckCommand) not set, or cannot find the specified program.
Errors while running CTest

I don’t know how to set CTEST_MEMORYCHECK_COMMAND, because sanitizers do not require any special command and documentation isn’t really helpful (https://cmake.org/cmake/help/latest/variable/CTEST_MEMORYCHECK_COMMAND.html#variable:CTEST_MEMORYCHECK_COMMAND). I expected that if CTEST_MEMORYCHECK_TYPE is set to one of sanitizers (https://cmake.org/cmake/help/latest/variable/CTEST_MEMORYCHECK_TYPE.html#variable:CTEST_MEMORYCHECK_TYPE) and sanitizer options are passed to CTEST_MEMORYCHECK_SANITIZER_OPTIONS, it should be enough to successful run tests.

Is there any documentation how to setup CTest with AddressSanitizer or any other sanitizer? I’ve searched for examples on GitHub, but none of them is really helpful. Most devs use just valgrind with ctest or set their on custom targets.

CMake version: 3.17
CTest command used: ctest --output-on-failure -D ExperimentalMemCheck
ASAN settings I’m using in CMakeLists.txt:

set(CTEST_MEMORYCHECK_TYPE "AddressSanitizer")
set(CTEST_MEMORYCHECK_SANITIZER_OPTIONS "verbosity=1:symbolize=1:abort_on_error=1:detect_leaks=1")
add_compile_options("-fsanitize=address")
add_link_options("-fsanitize=address")

Br,
Shaq

2

These only apply in CTest scripts; they have no effect in CMakeLists.txt. There, you need to use MEMORYCHECK_TYPE and MEMORYCHECK_SANITIZER_OPTIONS (as per the DartConfiguration.tcl.in template file).

3

Thanks for anwser. With MEMORYCHECK_TYPE and MEMORYCHECK_SANITIZER_OPTIONS in CMakeLists.txt I got same results:

Memory checker (MemoryCheckCommand) not set, or cannot find the specified program.

Errors while running CTest

Any other ideas?

4

What does the relevant section of your DartConfiguration.tcl look like?

5

my DartConfiguration.tcl looks like:

# Dynamic analysis (MemCheck)
PurifyCommand: 
ValgrindCommand: 
ValgrindCommandOptions: 
DrMemoryCommand: 
DrMemoryCommandOptions: 
MemoryCheckType: AddressSanitizer
MemoryCheckSanitizerOptions: verbosity=1:symbolize=1:abort_on_error=1:detect_leaks=1
MemoryCheckCommand: MEMORYCHECK_COMMAND-NOTFOUND
MemoryCheckCommandOptions: 
MemoryCheckSuppressionFile:
6

Hmm. Maybe just setting MEMORYCHECK_COMMAND to the empty string explicitly is necessary?

7

Hi,

With: set(MEMORYCHECK_COMMAND "")

DartConfiguration.tcl:

# Dynamic analysis (MemCheck)
PurifyCommand:
ValgrindCommand:
ValgrindCommandOptions:
DrMemoryCommand:
DrMemoryCommandOptions:
MemoryCheckType: AddressSanitizer
MemoryCheckSanitizerOptions: verbosity=1:symbolize=1:abort_on_error=1:detect_leaks=1
MemoryCheckCommand:
MemoryCheckCommandOptions:
MemoryCheckSuppressionFile:

But result is the same:

Memory checker (MemoryCheckCommand) not set, or cannot find the specified program.
Errors while running CTest

I can reproduce it on macOs and Linux.

8

Hmm. This seems like a defect in CMake. It shouldn’t require a memcheck command if an instrumented build is being used. It seems that I’m getting lucky where valgrind ends up being used as the command because it happens to be installed. Please file an issue (in the meantime, you can just set the command to /usr/bin/false or something.

9

Hi,

With set(MEMORYCHECK_COMMAND "/usr/bin/false") issue is still present.

DartConfiguration.tcl :

# Dynamic analysis (MemCheck)
PurifyCommand:
ValgrindCommand:
ValgrindCommandOptions:
DrMemoryCommand:
DrMemoryCommandOptions:
MemoryCheckType: AddressSanitizer
MemoryCheckSanitizerOptions: verbosity=1:symbolize=1:abort_on_error=1:detect_leaks=1
MemoryCheckCommand: /usr/bin/false
MemoryCheckCommandOptions:
MemoryCheckSuppressionFile:

/usr/bin/false exists:

$ ls /usr/bin/false
/usr/bin/false
$ /usr/bin/false
$ echo $? 
1

$ ctest --output-on-failure -D ExperimentalMemCheck                                                                                                                     ...
Memory checker (MemoryCheckCommand) not set, or cannot find the specified program.
Errors while running CTest

I’ve prepared minimal example: https://bitbucket.org/shaqmz/cmake_sanitizer/
Steps to reproduce:

  • mkdir build && cd build
  • cmake …
  • make
  • ctest --output-on-failure -D ExperimentalMemCheck
10

Any ideas worth trying to workaround CMake issue?

Bug report filed: https://gitlab.kitware.com/cmake/cmake/-/issues/20584

11

To use one of the sanitizers, set MEMORYCHECK_TYPE to one of the following strings (MEMORYCHECK_COMMAND will then be ignored):

  • AddressSanitizer
  • LeakSanitizer
  • MemorySanitizer
  • ThreadSanitizer
  • UndefinedBehaviorSanitizer

I have some recollection that you may also need to add the -fno-omit-frame-pointer compiler flag as well (at least in some cases), but I don’t recall the details.

Edit: Just saw your earlier reply where you said you tried this. If it isn’t working for you, it would be interesting if you could try an earlier CMake version. It’s possible there has been a regression.

12

@craig.scott @ben.boeckel to allow better understanding of problem I’ve setup CI verifivation like you’ve asked. I’ve tested latest minor version of all cmake major version starting from 3.1.3. With unset MEMORYCHECK_COMMAND, MEMORYCHECK_COMMAND set to "" and MEMORYCHECK_COMMAND set to "/usr/bin/false". Each job prints DartConfiguration.tcl content and performs test if /usr/bin/false exists in file system. Results are the same in all cases:

Memory checker (MemoryCheckCommand) not set, or cannot find the specified program.
Errors while running CTest

Please check CMakeLists.txt for obvious errors, but I couldn’t find any: https://bitbucket.org/shaqmz/cmake_sanitizer/src/master/CMakeLists.txt

If you have some additional tips what could be worth trying, please share anything.

repo: https://bitbucket.org/shaqmz/cmake_sanitizer/src/master/
CI jobs: https://travis-ci.com/bitbucket/shaqmz/cmake_sanitizer/builds/160373047

example jobs:
unset MEMORYCHECK_COMMAND: https://travis-ci.com/bitbucket/shaqmz/cmake_sanitizer/jobs/319589253
MEMORYCHECK_COMMAND set to “”: https://travis-ci.com/bitbucket/shaqmz/cmake_sanitizer/jobs/319589263
MEMORYCHECK_COMMAND set to dummy app: https://travis-ci.com/bitbucket/shaqmz/cmake_sanitizer/jobs/319589274

Additionally I’ve check CTest sources and couldn’t find regression test validating CTest behaviour with sanitizers, so I guess there is some mistake in CTest source code. I’ve tried simple code analysis with grep and find and couldn’t find “CMakeCommand” defined from https://gitlab.kitware.com/cmake/cmake/-/blob/master/Source/CTest/cmCTestMemCheckHandler.cxx line 493. Also other handlers are obtaining ActualCommand using other methods.

13

I made mistake in tests with dummy app. /usr/bin/false was not present in CI vm. I’ve repeated test with /bin/false instead and added check for availability of passed application on filesystem.

Results are the same :frowning:.

Changes made: hxxps://bitbucket.org/shaqmz/cmake_sanitizer/commits/6a0f6f7afb362fed27b3825d8a29a585cc37c806 (please change xx -> tt)

Results: https://travis-ci.com/bitbucket/shaqmz/cmake_sanitizer/builds/160390649

14

Since this currently looks like a bug, let’s keep discussion in the gitlab issue from this point rather than duplicating comments in both places. We can record any final outcome back here at the end once we resolve what’s happening.