Have a log file with a number increment every time ctest is run

Usage
1

Our group’s BASH driven test suite automatically increments the results to a log file:
run001.log and run001.sum
next would be
run002.log and run002.sum

I’m looking at converting the suite to run under CTest.
My initial thoughts were to use set(CTEST_CUSTOM_PRE_TEST fileBefore.py) and set(CTEST_CUSTOM_POST_TEST fileAfter.py) to run an appropriate python file before the test to find the latest number 002 and then increment to 003 for the next write. After the tests have been run to post-process all the results and assemble the summary and log files with the appropriate number (obtained from the pre step).
So, my questions:

  • Is this sensible or is there a better mechanism than set(CTEST_xxx afile.py) ? i.e. could this be controlled from within CMake/CTest?
  • Is there a way of the python returning a value back to CMake/CTest? I could write the 003 to a file and then have the after python pick that value up, increment to 004 and then do the post-processing of the results. However, having this value passed through the CTest seems a better solution.
2

Do you want a per-test log or a per-suite log?

If the latter, you could use a CTest script which calls ctest_start() which makes a datestamp-based directory under Testing/Temporary which stores the suite-level logs. It’d mean that ctest and make test wouldn’t work as intended, but ctest -S test.cmake would be what you’d want (though you could make a custom target to do this for you such as make ctest or make check instead).

For the former, I’d consider adding log_increment.py as a wrapper to each test command which then passes on the intended log number to the test as an additional argument:

add_executable(mytestexe …)
add_test(TEST foo COMMAND log_inc.py $<TARGET_FILE:mytestexe> arg1 arg2)

The $<TARGET_FILE> is needed because CMake only does the automatic replacement if the executable name is the first in the command argument list.

3

For information:
I ended up using the second method -
set(CTEST_CUSTOM_PRE_TEST beforeTest.py) and set(CTEST_CUSTOM_POST_TEST "${CMAKE_SOURCE_DIR}/afterTest.py ) to set up my counter from previous logs. A custom python script to run the executable we’re testing and the afterTest.py to gather and post-process the results.