# Unexpected/inconsistent behavior of cmake -E compare\_files

**URL:** <https://discourse.cmake.org/t/unexpected-inconsistent-behavior-of-cmake-e-compare-files/1680>\
**Category:** Code\
**Created:** [August 9, 2020, 11:57pm UTC](https://discourse.cmake.org/t/unexpected-inconsistent-behavior-of-cmake-e-compare-files/1680 "2020-08-09T23:57:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![arclight](https://discourse.cmake.org/user_avatar/discourse.cmake.org/arclight/32/785_2.png) [@arclight](https://discourse.cmake.org/u/arclight)\
**Post date:** [August 9, 2020, 11:57pm UTC](https://discourse.cmake.org/t/unexpected-inconsistent-behavior-of-cmake-e-compare-files/1680/1 "2020-08-09T23:57:07Z")

</div>

I’m attempting to compare code output with reference text using CTest (`add_test()`) and I’m getting some strange results which makes me think I’m not using `cmake -E compare_files` correctly. The documentation and examples are a bit thin and I haven’t found an example similar to what I’m trying to do.

Backstory: I’m recovering some creaky ancient FORTRAN code which expects input via `stdin` and writes output to `stdout`. Knowing that I/O redirection in `add_test()` is discouraged/impossible, I use `add_custom_command()` as a `POST_BUILD` action on the executable target to run the code and redirect output. That works as expected - no problems there.

I then set up two tests resembling:

```auto
add_test(NAME C1_bad
    COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol C1.OUT C1_bad.OUT
    CONFIGURATIONS Debug Release ""
    WORKING_DIR "${MY_TEST_DIR}"
)

# Compare output and unverified recent example output
add_test(NAME C1_ok
    COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol C1.OUT C1_ok.OUT
    CONFIGURATIONS Debug Release ""
    WORKING_DIR "${MY_TEST_DIR}"
)

```

`${MY_TEST_DIR}/C1.OUT` is generated by the post-build custom command - this is fresh output. `${MY_TEST_DIR}/C1_bad.OUT` is copied from the source tree and is different from `C1.OUT`. `${MY_TEST_DIR}/C1_ok.OUT` is copied from the source tree and is binary identical to `C1.OUT` (confirmed by inspection and MD5 checksum).

Unexpectedly, both tests fail, not just the first one. Running `cmake -E compare_files` from the command line against each pair of files gives expected results - comparing `C1_bad.OUT` and `C1.OUT` fails but comparing `C1_ok.OUT` and `C1.OUT` succeeds.

My thought is that `cmake -E compare_files` does not return status codes expected by CTest (0 to pass, non-zero to fail) or is otherwise not intended for use with CTest. This is with version 3.18.1 on Windows if that matters. I don’t want to chase down or write my own cross-platform diff utility if CMake has one already built in but it’s not clear from everything I’ve read whether this is an expected or workable use of `compare_files` - any hints on what I should try next?

---

<div class="post-metadata">

**Author:** ![arclight](https://discourse.cmake.org/user_avatar/discourse.cmake.org/arclight/32/785_2.png) [@arclight](https://discourse.cmake.org/u/arclight)\
**Post date:** [August 13, 2020, 2:38pm UTC](https://discourse.cmake.org/t/unexpected-inconsistent-behavior-of-cmake-e-compare-files/1680/2 "2020-08-13T14:38:37Z")

</div>

I have more information about this issue. If I change the tests to use full paths in `compare_files` instead of bare filenames and remove the `WORKING_DIR` property from `add_test`, the tests behave as expected. This would seem to indicate that `WORKING_DIR` does not carry into the `compare_files` command. Changing the tests to the following gives the expected behavior:

```auto
add_test(NAME C1_bad
    COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol "${MY_TEST_DIR}/C1.OUT" "${MY_TEST_DIR}/C1_bad.OUT"
    CONFIGURATIONS Debug Release ""
)

add_test(NAME C1_ok
    COMMAND ${CMAKE_COMMAND} -E compare_files --ignore-eol "${MY_TEST_DIR}/C1.OUT" "${MY_TEST_DIR}/C1_ok.OUT"
    CONFIGURATIONS Debug Release ""
)

```

This fixes my immediate issue so my question is now - _is this expected behavior?_ If so, where is it documented? If not, should this be filed as a bug?

---

<div class="post-metadata">

**Author:** ![fdk17](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/f/ea666f/32.png) [@fdk17](https://discourse.cmake.org/u/fdk17)\
**Post date:** [August 13, 2020, 3:49pm UTC](https://discourse.cmake.org/t/unexpected-inconsistent-behavior-of-cmake-e-compare-files/1680/3 "2020-08-13T15:49:37Z")

</div>

So what you are saying is that `add_test` ignores `WORKING_DIR`? But the documentation says that the parameter should be `WORKING_DIRECTORY`. Since the wrong keyword is being used won’t your `add_test` command interpret `WORKING_DIR` and `"${MY_TEST_DIR}"` as additional `CONFIGURATIONS`?

---

<div class="post-metadata">

**Author:** ![arclight](https://discourse.cmake.org/user_avatar/discourse.cmake.org/arclight/32/785_2.png) [@arclight](https://discourse.cmake.org/u/arclight)\
**Post date:** [August 13, 2020, 5:59pm UTC](https://discourse.cmake.org/t/unexpected-inconsistent-behavior-of-cmake-e-compare-files/1680/4 "2020-08-13T17:59:06Z")

</div>

Thanks @fdk17! I’m going to feel really stupid if it’s that simple of a typo…

(Narrator: _“It’s that simple of a typo.”_)

Further, if it’s a typo, why isn’t CMake crashing out on unidentified/unknown property to `add_test()`?

---

<div class="post-metadata">

**Author:** ![Angew](https://discourse.cmake.org/user_avatar/discourse.cmake.org/angew/32/229_2.png) [@Angew](https://discourse.cmake.org/u/Angew)\
**Post date:** [August 17, 2020, 6:06pm UTC](https://discourse.cmake.org/t/unexpected-inconsistent-behavior-of-cmake-e-compare-files/1680/5 "2020-08-17T18:06:25Z")

</div>

> [@arclight](#):
>
> Further, if it’s a typo, why isn’t CMake crashing out on unidentified/unknown property to `add_test()` ?

As @fdk17 pointed out, it’s simply interpreting the strings `WORKING_DIR` and whatever comes from the expansion of `MY_TEST_DIR` as additional configuration names, since they follow the multi-values keyword argument `CONFIGURATIONS`.