duplication of test execution with ctest_memcheck() and sanitizers

Usage
1

Hi,

We use CTest to drive CMake to run unit tests which get published on CDash.

Inspired by the blog post in [1] we aim at the following:

  • run unit tests with memory checks (in our case AddressSanitizer)
  • publish any memory issues/leaks/etc in Dynamic Analysis section of CDash

So our CTest file includes the relevant calls:

set ( CTEST_MEMORYCHECK_TYPE "AddressSanitizer" )
ctest_memcheck ()

But I observe in the logs that each of our unit tests is run twice. Once as a Test command: and once as a MemCheck command:.

Example from logs (NOTE: This is on macOS):

1: MemCheck command: /Applications/CMake.app/Contents/bin/cmake "-E" "env" "ASAN_OPTIONS=log_path='/Users/builder/rogue_nightly_build/scratch/10.15-UnitTests-trunk-ASan-Debug-bin/Testing/Temporary/MemoryChecker.1.log'" "/Applications/Xcode.app/Contents/Developer/usr/bin/xctest" "-XCTest" "UT_RRFoo/testBar" "/Users/builder/somewhere/UnitTests.xctest"
1: Test command:     /Applications/Xcode.app/Contents/Developer/usr/bin/xctest  "-XCTest" "UT_RRFoo/testBar" "/Users/builder/somewhere/UnitTests.xctest"

It seems the difference between the two invocations is just the ASAN_OPTIONS environment variable with a logging path.

Is there a way to run each unit tests just once?
(I can see how for some memcheck tools running twice make sense, but not sure it is needed for instrumented test like when using AddressSanitizer).

In our case this would save many CPU-hours on a daily basis on each build machine that runs our tests.

sincerely,
Alex Ciobanu

[1] CTest and CDash add support for new dynamic analysis tools - Kitware Blog

3

Hmm. I’m not seeing that with one of our projects. We’re not using XCTest at all though. I wonder if the Xcode generator is getting involved here somehow?

4

Hi Ben,

Your message made me realize that we were using the following sequence in our CTest file (as done in the blog post):

ctest_configure()
ctest_build()
ctest_test()
ctest_memcheck()

but in fact the ctest_test() is optional, which I thought it was required for ctest_memcheck().

I am now able to just run the MemCheck-type tests/invocations:

ctest_configure()
ctest_build()
ctest_memcheck()

Thanks!