Unable to set ctest specific property for parameterized gtest tests

Usage
1

Hi!
I have parameterized GTest tests and struggle to set CTest properties on them. The problems seems to be the parameter type. If I have tests with string parameters, everything is fine. As soon as the parameter type is a custom type, things start to break. The problem is: CTest seems to be unable to identify the tests.
As soon as the test is based on a custom type parameter, the name has such a cryptic name at the end 4-byte object <02-00 00-00>. If the parameter type is just string, then it is "foo".

Here are a few concrete examples:

#include <gmock/gmock.h>
#include <gtest/gtest.h>

using namespace testing;

class TestFixture : public TestWithParam<std::string>
{
};

TEST_P(TestFixture, test_enabled)
{
    std::cout << "param: " << GetParam() << "\n";
    SUCCEED();
}

TEST_P(TestFixture, test_disabled)
{
    std::cout << "param: " << GetParam() << "\n";
    FAIL();
}

INSTANTIATE_TEST_SUITE_P(DifferentStrings, TestFixture, testing::Values("foo", "bar"));

I’m able to to disable specific tests in CMake with the following snippet:

set_tests_properties(
    DifferentStrings/TestFixture.test_b/"foo"
	DifferentStrings/TestFixture.test_b/"bar"

    PROPERTIES
        DISABLED ON
)

Here is a snippet of a test that CTest fails to identify:

struct Foo
{
    int x;
};

class ParameterizedTestFixture : public TestWithParam<Foo>
{
};

TEST_P(ParameterizedTestFixture, test_a)
{
    std::cout << "param: " << GetParam().x << "\n";
    SUCCEED();
}

TEST_P(ParameterizedTestFixture, test_b)
{
    std::cout << "param: " << GetParam().x << "\n";
    FAIL();
}

INSTANTIATE_TEST_SUITE_P(DifferentObjects, ParameterizedTestFixture, testing::Values(Foo{1}, Foo{2}));

GTest is logging the test names as DifferentStrings/TestFixture.test_b/4-byte object <02-00 00-00>. Using this as a test identifier in CTest does unfortunately not work.

set_tests_properties(
	DifferentStrings/ParameterizedTestFixture.test_b/4-byte object <01-00 00-00>
    DifferentStrings/ParameterizedTestFixture.test_b/4-byte object <02-00 00-00>

    PROPERTIES
        DISABLED ON
)

Because of reasons I need to use the gtest_discover_tests way, provided from GTest, to register tests with CTest.

Any ideas what could be the problem? Thx!

2

Put quotes around the test names in your call to set_tests_properties().

set_tests_properties(
	"DifferentStrings/ParameterizedTestFixture.test_b/4-byte object <01-00 00-00>"
    "DifferentStrings/ParameterizedTestFixture.test_b/4-byte object <02-00 00-00>"

    PROPERTIES
        DISABLED ON
)

I assume you are calling set_tests_properties() inside a custom ctest script that you add to the TEST_INCLUDE_FILES directory property?

1 Like
3

I feel so dumb for not trying this… Thank you!

A follow up question would be if wildcards are supported as well in any way. It’s nice to be able to set properties for each test method individually, but sometimes it would be handy to be able to write

set_tests_properties(
    "DifferentStrings/ParameterizedTestFixture.*"
)

or for non parameterized tests

set_tests_properties(
    "FooTest.*"
)

So far I haven’t found an answer to this and it does not work.

Thx!

4

Wildcards are not supported, you need to list all properties by their exact name. But check the TEST_LIST keyword of gtest_discover_tests(). You can use it to have that command populate a variable you nominate and you can read that variable in the custom ctest script. You can apply whatever filtering you want to that list and then call set_tests_properties() on the filtered list, so you can indirectly get the behavior you want that way.

1 Like
5

Ah, just found this topic; @craig.scott you are pointing in the direction of the answer I’m searching for. Obviously the TEST_LIST variable is unavailable before the test executable has been built, so it’s still empty immediately after the call to gtest_discovered_tests(), and I need to do something else, but I don’t know about custom ctest scripts. GoogleTest itself is clearly generating some scripts. Do I need to follow the same pattern? I guess it’s the directory’s TEST_INCLUDE_FILES property that causes these scripts to get used. Just confirming I’ve got all that right.

Whew, that all should really be documented.

6

You can’t access the list of tests from your CMake project during the configure run. TEST_LIST is naming a variable you can access during ctest execution. The querying of the tests only happens at test time (or maybe at build time, depending on how you define things). You would access the variable named by TEST_LIST from a custom script that you write and tell ctest about via the TEST_INCLUDE_FILES directory property.

I’d welcome a merge request that improves the documentation around this.