Unit tests failing only when run via CTest

Usage
1

Have the following situation:

  1. C++ codebase
  2. Unit test suite based on the Boot.Test
  3. When run directly all the tests pass no issues
  4. When run via ctest eg: ctest -j 1 or ctest -j 5 tests fail
  5. It is the same test every time
  6. Only happens with a specific C++ compiler (clang6) and only in release builds
  7. Same codebase and UTs built with other compilers GCC and other versions of clang work fine
  8. The UTs don’t use shared memory or environment variables etc.
  9. Happens with or without –force-new-ctest-process

Q1: Has anyone encountered a situation like this before?

Q2: How would one go about investigating whats going on?

2

I would run that test with verbosity (ctest -R '$testname' -VV) to see how CTest is invoking it. Things to note:

  • CTest may be setting extra environment settings (via the ENVIRONMENT property)
  • Clang could have a bug (6 is pretty old these days); release-only points to something optimizer-related

For debugging, I would run the test inside and outside of CTest using valgrind to see if it finds anything (you’ll need to modify the add_test to have it run under CTest). You can also try the sanitizers (ASan and UBSan are of particular note), though Clang 6’s sanitizers may be less useful than new ones (that said, new ones could also point the problem out without crashing too).