I’m attempting to compare code output with reference text using CTest (add_test()
) and I’m getting some strange results which makes me think I’m not using cmake -E compare_files
correctly. The documentation and examples are a bit thin and I haven’t found an example similar to what I’m trying to do.
Backstory: I’m recovering some creaky ancient FORTRAN code which expects input via stdin
and writes output to stdout
. Knowing that I/O redirection in add_test()
is discouraged/impossible, I use add_custom_command()
as a POST_BUILD
action on the executable target to run the code and redirect output. That works as expected - no problems there.
I then set up two tests resembling:
add_test(NAME C1_bad
COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol C1.OUT C1_bad.OUT
CONFIGURATIONS Debug Release ""
WORKING_DIR "${MY_TEST_DIR}"
)
# Compare output and unverified recent example output
add_test(NAME C1_ok
COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol C1.OUT C1_ok.OUT
CONFIGURATIONS Debug Release ""
WORKING_DIR "${MY_TEST_DIR}"
)
${MY_TEST_DIR}/C1.OUT
is generated by the post-build custom command - this is fresh output. ${MY_TEST_DIR}/C1_bad.OUT
is copied from the source tree and is different from C1.OUT
. ${MY_TEST_DIR}/C1_ok.OUT
is copied from the source tree and is binary identical to C1.OUT
(confirmed by inspection and MD5 checksum).
Unexpectedly, both tests fail, not just the first one. Running cmake -E compare_files
from the command line against each pair of files gives expected results - comparing C1_bad.OUT
and C1.OUT
fails but comparing C1_ok.OUT
and C1.OUT
succeeds.
My thought is that cmake -E compare_files
does not return status codes expected by CTest (0 to pass, non-zero to fail) or is otherwise not intended for use with CTest. This is with version 3.18.1 on Windows if that matters. I don’t want to chase down or write my own cross-platform diff utility if CMake has one already built in but it’s not clear from everything I’ve read whether this is an expected or workable use of compare_files
- any hints on what I should try next?