vs test explorer and gtest_discover_test working directory

Usage
1

Hi! I have a strange issue with gtest_discovery_tests and custom working directory with VS Test Explorer – on some machines the working directory is set as expected and on some it doesn’t.

In my project I have a tests directory with subdirectories for each unit test suite:

CMakeLists.txt
|-- tests/
     |-- CMakeLists.txt
     |-- data/
     |   | -- CMakeLists.txt
     |-- unit1/
         |-- CMakeLists.txt
...

Root CMakeLists.txt contains enable_testing() and add_subdirectory(tests). In the tests/CMakeLists.txt I reset runtime output directory to the current binary dir: set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}).

Each unit’s CMakeLists.txt contains gtest_discover_tests(tst_unit WORKING_DIRECTORY ${CMAKE_RUNTIME_OUTPUT_DIRECTORY})

There is also a data project which contains some external files for unit tests. Its CMakeLists.txt just copies them into runtime directory:
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/data/" DESTINATION "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/data").

The problem is that for several of my colleagues the working directory seems not set and the default one is used when run the tests from VS Test Explorer. And running the tests using ctest command works for them. The test files are copied to the correct directory while test executables are placed in their own subdirectories instead of the common one I set in the tests/CMakeLists.txt

I’m stuck as on my machine everything works. Did I misconfigure or forget to set something? Maybe it’s not a good idea to reset CMAKE_*_DIRECTORY variables and better to use custom ones? Or it’s not cmake issue at all and I should ask VS guys?

cmake 3.30.0, windows 11, vs 2020 17.10.4

2

My first thought is to check the directory you’re trying to set WORKING_DIRECTORY to. For the machines that fail, does that path contain any spaces or other special characters? It’s possible that the way the path is copied to some scripts is not handling those correctly, but let’s see if there’s some common characteristic to the paths that are failing compared to those that don’t.

3

I haven’t noticed any special characters in the paths.
CMakeCache.2.txt (44.7 KB)
CMakeCache.txt (52.9 KB)

4

Strange indeed. I don’t know how Visual Studio’s Test Explorer runs the tests, but it seems suspicious that your users only see problems when running tests through that. If it is working from the command line, that makes me think there’s something unusual in how it is doing that. I can’t offer any further advice on this one except maybe post a bug report in their issue tracker and see if they give you any response. The fact that running ctest directly works for all your users makes it less likely to be a bug in CMake/ctest.

Maybe also check what Visual Studio version people are running and see if there is a correlation between the version and the machines where it fails.

5

Thanks for the reply, Craig! I suspected it’s a VS issue, but just wanted to be sure my approach with cmake is generally correct. I’ll ask VS guys and update this topic if a solution is found.

6

I had the same issue, but couldn’t get it to work.
I ended up changing our tests to be able to run in the target directory.
They needed access to some rather large data files that were to cumbersome to copy to the target directory each time, so I had CMake generate a file with paths to the data files next to the test executable. The tests read this file to find the location of the data files.

7

Another thing to check is what ctest Visual Studio uses when it runs tests. In the past, Microsoft have bundled their own non-official CMake as part of the VS install. If you can track down what ctest they are using, it would be worthwhile trying to run that particular binary from the command line to see if you can reproduce your problem that way. If you can, that would point to a change they made rather than a bug in CMake itself.

8

It seems that the problem was with data library, it’s header-only and VS puts it under <builddir>/tests while the rest units under <builddir>/tests/<BuildType>. One way to workaround this is to make data a normal library thus it will be placed together with the rest tests. But it doesn’t explain why the original approach works on some machines and not on the others.

I gave up to investigate this and started to use the absolute names for the external files in data library. In the data/CMakeLists.txt I added:

set(DATA_DIR "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/data")
target_compile_definitions(data INTERFACE DATA_DIRECTORY=${DATA_DIR})

and in data/data.h I added:

#define TEST_STRINGIFY(x) #x
#define TEST_STRINGIFY_DEFINE(x) TEST_STRINGIFY(x)

static inline const std::string current_test_dir = TEST_STRINGIFY_DEFINE(DATA_DIRECTORY);

// My colleagues and me used both, VS’ cmake and the standalone one, the results were the same – works for me, doesn’t work for them.

9

Do you and your colleagues have the same Visual Studio version installed?

10

No, one of them has vs19 while the rest have vs22

11

You need to look at the full version, not just v19 or v22. Check what update/patch release is installed. It is not unusual for a problem to exist in one update and be fixed in another within a major release series.