CTest execution much slower than running the executable

I’ve got some issues with my tests, when running with CTest. The tests are written using GTest, in particular there is a parametrized test case with about a hundred cases. But the same holds for other test (cases) as well.

When I start the test executable from command-line or IDE (Qt-creator), the test (all cases instantiations executed) takes about 93ms.
When I start the test using CTest, it takes 2 seconds each , so more than 200s in total.

How can that be?

And, even more important: How can I fix that?

Definition is as follows:

    add_executable(myTest WIN32 testCases.cpp)
    target_compile_options(myTest PRIVATE -Wall -Wextra -Wpedantic -Werror)
    target_link_libraries(myTest PUBLIC gtest_main)
    gtest_discover_tests(myTest)

The test itself is similar to this:

#include <QVariant>

#include <gtest/gtest.h>

struct test_data
{
    QString typeName;
    QVariant input;
};

class MyTestValues: public testing::TestWithParam<test_data>
{
};

TEST_P(MyTestValues, properties)
{
    auto values = GetParam();
    // do some testing
}

INSTANTIATE_TEST_SUITE_P(
    MyTest,
    MyTestValues,
    testing::Values(
        test_data{ QString{ "bool" }, QVariant{ true } },
        test_data{ QString{ "bool" }, QVariant{ false } },
        test_data{ QString{ "char" }, QVariant::fromValue(std::numeric_limits<char>::min()) },
        test_data{ QString{ "char" }, QVariant::fromValue(std::numeric_limits<char>::max()) },
        test_data{ QString{ "char" }, QVariant::fromValue(static_cast<char>(0)) },
        test_data{ QString{ "int" }, QVariant{ 42 } },
        test_data{ QString{ "int" }, QVariant{ std::numeric_limits<int>::min() } },
        test_data{ QString{ "int" }, QVariant{ std::numeric_limits<int>::max() } },
        test_data{ QString{ "long" }, QVariant::fromValue(static_cast<long>(-1)) },
        test_data{ QString{ "long" }, QVariant::fromValue(std::numeric_limits<long>::min()) },
        test_data{ QString{ "long" }, QVariant::fromValue(std::numeric_limits<long>::max()) }
// omitting many other values
));

IIUC, GTest tests are run one at a time by CTest, so it ends up launching the executable for each individual test. There may be a way to run them as a single test, but then you lose granularity at the CTest level. You can also investigate if there is unnecessary setup in the one-test-at-a-time setup that you can reduce.

I expected some overhead from running the tests one by one by CTest. I’m aware, that CTest starts the test executable for each test-case explicitly, if configured with gtest_discover_tests.

But the values I have here are ridiculous. One of the single test cases executed with CTest takes 20 times the time all test cases take if run in one go.

I’ve been working for years with CMake/CTest on Linux and never have never seen such numbers. I searched through the CMake files for the project definition, but there is nothing out of the ordinary, I don’t even use pre-test-scripts.

Running a single one of these tests gives me this output:

Note: Google Test filter = MyTest/MyTestValues.properties/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from MyTest/MyTestValues
[ RUN      ] MyTest/MyTestValues.properties/0
[       OK ] MyTest/MyTestValues.properties/0 (0 ms)
[----------] 1 test from MyTest/MyTestValues (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (0 ms total)
[  PASSED  ] 1 test.

It cannot even record the time it took to execute that test.
CTest gives me this output:

29: Test timeout computed to be: 1500
29: Running main() from C:/path/to/build/directory/_deps/googletest-src/googletest/src/gtest_main.cc
29: Note: Google Test filter = MyTest/MyTestValues.properties/0
29: [==========] Running 1 test from 1 test suite.
29: [----------] Global test environment set-up.
29: [----------] 1 test from MyTest/MyTestValues
29: [ RUN      ] MyTest/MyTestValues.properties/0
29: [       OK ] MyTest/MyTestValues.properties/0 (0 ms)
29: [----------] 1 test from MyTest/MyTestValues (0 ms total)
29:
29: [----------] Global test environment tear-down
29: [==========] 1 test from 1 test suite ran. (0 ms total)
29: [  PASSED  ] 1 test.
  1/124 Test  #29: MyTest/MyTestValues.properties/24-byteobject<80-A88F-0200-0000-0001-2500-0000-0000-0001-0000-0000-0000-00> ...   Passed    1.76 sec

GTest still tells me, that the test execution took almost no time, so where do the 1.76 sec CTest takes to execute it come from?

It comes mostly from the overhead of starting a new process on Windows (yes, it’s that bad…).

:sob:

I already regret taking on that new job