Is there a reason why I should expect CTest to run tests from within a Google Test executable much more slowly than the command line or Visual Studio Test Explorer?
Give an test suite with 744 tests inside a single executable compiled with Google Test, in which there are no filesystem or network interactions:
-
Just running the executable takes 3.24 s, with the longest individual test taking 3ms to run.
-
Running all the tests from within Visual Studio 2019 Test Explorer takes 1.6 seconds.
-
Running the tests from CTest (by just navigating to the binary directory and typing “ctest”) takes 160 seconds (i.e. a full 100X as long as Test Explorer), with every test appearing to take either 0.20 or 0.21 seconds.
What’s the cause of the overhead involved in CTest? Can it be mitigated? As the number of tests in our project grows, it will be hard to justify using CTest if this behaviour is typical.
When run through ctest, do you see one test case or 744? If the latter, then the test executable is being invoked 744 times instead of just once. On Windows, the cost of starting a process is non-trivial and likely explains the difference (although it’s a pretty big difference even then).
You can potentially mitigate some of that by running tests in parallel, which you can do by passing a -j option to ctest or setting the CTEST_PARALLEL_LEVEL environment variable before invoking ctest.
As for why ctest might do this, note that it isn’t required to do it this way. It depends on how the project is defining the tests. If it uses the GoogleTest module and calls commands like gtest_add_tests() or gtest_discover_tests(), then that’s why there are many individual test cases being created. The advantage is that you see the result of each individual test case in the output, which can be very useful if the test run is reporting its results to CDash or similar. It’s a tradeoff - more granular output but more overhead per test case.
Thank you.
Yes, there are 744 separate test cases so no doubt that’s what’s happening. Running the tests with -j4 on a four-core Intel i5 roughly halves the total time taken (since each individual test seems to take twice as long, ~0.4s rather than ~0.2s).
OK, I’m going to need to think carefully about whether this is the way we should be doing things.