Cannot set property of a test in a subdirectory

Code
1

I’m having trouble setting properties on all tests in a directory tree.

Here is a dummy project that illustrates the issue

cmake_minimum_required(VERSION 3.17)
project(Hello_world VERSION 1.0)
enable_testing()

# get_all_tests  will walk from CMAKE_CURRENT_SOURCE_DIR & find all tests
function(get_all_tests _out_var)
    set(${_out_var} "")
    get_all_tests_walker(${_out_var} ${CMAKE_CURRENT_SOURCE_DIR})

    set(${_out_var} "${${_out_var}}" PARENT_SCOPE)

endfunction()

function(get_all_tests_walker _out_var _directory)

    get_property(_subdirs DIRECTORY "${_directory}" PROPERTY SUBDIRECTORIES)

    foreach(_subdir IN LISTS _subdirs)
        get_all_tests_walker(${_out_var} "${_subdir}")
    endforeach()

    get_directory_property(_tests DIRECTORY "${_directory}" TESTS)
    set(${_out_var} ${${_out_var}} ${_tests} PARENT_SCOPE)

endfunction()

# add a subdir with a test
add_subdirectory(another_dir)

# add a test local to the current directory

add_test(
    NAME localtest
    COMMAND localtest
)

# read all test names into "alltests"
get_all_tests(alltests)

message(WARNING "${alltests}")

set_property(TEST ${alltests}
        APPEND
        PROPERTY
            LABELS PROTOBUF mylabel
)

The directory another-dir has this

add_test(
    NAME subdirtest
    COMMAND mycommand
)

When I run cmake against the above I get this output

$ cmake  ..
CMake Warning at CMakeLists.txt:126 (message):
  subdirtest;localtest


CMake Error at CMakeLists.txt:141 (set_property):
  set_property given TEST names that do not exist:

    subdirtest

-- Configuring incomplete, errors occurred!

Observations

  1. the call to get_all_tests has correctly found all the tests in the directory tree

  2. the call to set_property cannot find the subdirtest test
    It appears that set_property(TEST only sees tests in the current directory

Is this defined behaviour or have I missed something obvious?

2

It does indeed seem that setting a test property only looks for the test in the current directory scope. The docs should be improved to mention this.

3

Is the current behaviour of limiting the scope to the current directory the intended behaviour or is it a bug/oversight in the current implementation?

If this is defined behaviour, can I raise enhancement request?

4

It seems to be intended. Modifying tests in other directories is certainly possible, but it doesn’t look trivial (the directory scope tree would need walked to find the test, though whatever mechanism guarantees test name uniqueness could be reused (and enhanced if necessary) to do a more direct lookup.

Sure, file an issue. No guarantees on any timeline to actually changing it (though MRs are certainly welcome).

5

As far as I’m aware, test names are not guaranteed nor required to be unique beyond the directory scope in which they are defined, possibly not even within a directory scope. Even CMake’s own test suite uses duplicated test names in some places. (sorry, had to edit my original statement after realising it said almost the opposite of what I intended)

6

Yep - just tried that out - within a subdirectory all test names need to be unique, but once you span multiple directories you can have duplicates. Damn!

That then suggests that the technique of walking a directory tree to collect all the tests using the existing implementation of get_directory_property(...TESTS) is never going to work, even if set_property(TEST was somehow enhanced to work across directories. You can never guarantee that the list of tests is complete if there are potential duplicates.

Are there any other techniques available that would allow test properties to be updated that don’t just involve making the change at the point where the call to add_test is made, like below?

add_test(
    NAME localtest
    COMMAND some command
)

set_property(
    TEST
        localtest
    APPEND PROPERTY
        ENVIRONMENT FRED=${FRED}
)
7

A function can do it (it has a different variable scope, but inherits the directory scope from the caller). So something like:

add_test(NAME blah)
apply_my_test_props(blah)
8

That doesn’t help because I need to span directory scopes.

Let me rephrase my question – are there any other techniques available that would allow test properties from a different directory scope to be updated?

9

Not that I’m aware of, no.

10

OK. I’ve created issue https://gitlab.kitware.com/cmake/cmake/-/issues/22813 to record the issue anyway

11

Related on Stack Overflow: Is it possible in CMake to set properties of tests in a different directory scope than the one the test is defined in?