Unexpected/inconsistent behavior of cmake -E compare_files

Code
1

I’m attempting to compare code output with reference text using CTest (add_test()) and I’m getting some strange results which makes me think I’m not using cmake -E compare_files correctly. The documentation and examples are a bit thin and I haven’t found an example similar to what I’m trying to do.

Backstory: I’m recovering some creaky ancient FORTRAN code which expects input via stdin and writes output to stdout. Knowing that I/O redirection in add_test() is discouraged/impossible, I use add_custom_command() as a POST_BUILD action on the executable target to run the code and redirect output. That works as expected - no problems there.

I then set up two tests resembling:

add_test(NAME C1_bad
    COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol C1.OUT C1_bad.OUT
    CONFIGURATIONS Debug Release ""
    WORKING_DIR "${MY_TEST_DIR}"
)

# Compare output and unverified recent example output
add_test(NAME C1_ok
    COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol C1.OUT C1_ok.OUT
    CONFIGURATIONS Debug Release ""
    WORKING_DIR "${MY_TEST_DIR}"
)

${MY_TEST_DIR}/C1.OUT is generated by the post-build custom command - this is fresh output. ${MY_TEST_DIR}/C1_bad.OUT is copied from the source tree and is different from C1.OUT. ${MY_TEST_DIR}/C1_ok.OUT is copied from the source tree and is binary identical to C1.OUT (confirmed by inspection and MD5 checksum).

Unexpectedly, both tests fail, not just the first one. Running cmake -E compare_files from the command line against each pair of files gives expected results - comparing C1_bad.OUT and C1.OUT fails but comparing C1_ok.OUT and C1.OUT succeeds.

My thought is that cmake -E compare_files does not return status codes expected by CTest (0 to pass, non-zero to fail) or is otherwise not intended for use with CTest. This is with version 3.18.1 on Windows if that matters. I don’t want to chase down or write my own cross-platform diff utility if CMake has one already built in but it’s not clear from everything I’ve read whether this is an expected or workable use of compare_files - any hints on what I should try next?

2

I have more information about this issue. If I change the tests to use full paths in compare_files instead of bare filenames and remove the WORKING_DIR property from add_test, the tests behave as expected. This would seem to indicate that WORKING_DIR does not carry into the compare_files command. Changing the tests to the following gives the expected behavior:

add_test(NAME C1_bad
    COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol "${MY_TEST_DIR}/C1.OUT" "${MY_TEST_DIR}/C1_bad.OUT"
    CONFIGURATIONS Debug Release ""
)

add_test(NAME C1_ok
    COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol "${MY_TEST_DIR}/C1.OUT" "${MY_TEST_DIR}/C1_ok.OUT"
    CONFIGURATIONS Debug Release ""
)

This fixes my immediate issue so my question is now - is this expected behavior? If so, where is it documented? If not, should this be filed as a bug?

3

So what you are saying is that add_test ignores WORKING_DIR? But the documentation says that the parameter should be WORKING_DIRECTORY. Since the wrong keyword is being used won’t your add_test command interpret WORKING_DIR and "${MY_TEST_DIR}" as additional CONFIGURATIONS?

1 Like
4

Thanks @fdk17! I’m going to feel really stupid if it’s that simple of a typo…

(Narrator: “It’s that simple of a typo.”)

Further, if it’s a typo, why isn’t CMake crashing out on unidentified/unknown property to add_test()?

5

As @fdk17 pointed out, it’s simply interpreting the strings WORKING_DIR and whatever comes from the expansion of MY_TEST_DIR as additional configuration names, since they follow the multi-values keyword argument CONFIGURATIONS.