Feature Request - Allow PROGRAM and ARGS fields in add_test

Development
1

I ran into a problem when trying to debug a test on a remote host using vscode. I discussed the details of the issue in the code section, Problem debugging test on remote host.

The issue revolves around the fact that the vs-cmake-tool extension to vscode expects the file containing the symbol table to be defined in the first element of the COMMAND field. I had to make the first command be a script to set up the remote host to run the executable. When I try to debug the test, it fails . It looks at the script to get the symbol table which fails.

In vs-cmake-tool the definition of testProgram happens here in ctests.ts:

 private testProgram(testName: string): string {
        if (this.tests) {
            for (const test of this.tests.tests) {
                if (test.name === testName) {
                    return test.command[0];
                }
            }
        } else if (this.legacyTests) {
            for (const test of this.legacyTests) {
                if (test.name === testName) {
                    return test.name;
                }
            }
        }
        return '';

It really is the best choice to assign testProgram to command[0] with the information that is available from CMake.

It would be nice to add a new field to add_test that allows it to specifically provide the program to use to gather the symbol table.

I would like to add a PROGRAM and ARGS field to add_test(). This would allow vs-cmake-tools to define it’s testProgram as PROGRAM if it exists. Otherwise, go ahead and use command[0]. I have no experience with typescript. I believe the vs-cmake-tools code would look like this:

private testProgram(testName: string): string {
        if (this.tests) {
            for (const test of this.tests.tests) {
                if (test.name === testName) {
                    const property = test?.properties
                         .find(prop => prop.name === 'PROGRAM');
                    if (typeof (property?.value) === 'string') {
                        return property.value;
                    }
                    return test.command[0];
                }
            }
        } else if (this.legacyTests) {
            for (const test of this.legacyTests) {
                if (test.name === testName) {
                    return test.name;
                }
            }
        }
        return '';
    }

We have a similar issue for vs-cmake-tools defintion of testArgs. It has to parse COMMAND and it assumes the args are all the values after the first element of COMMAND. Adding an ARGS field to add_test() allows testArgs to be set independent of COMMAND.

I have made changes to cmAddTestCommand.cxx to provide these extra properties. The changes really have no impact on cmake directly. They just carry these properties along and make them available to tools like vs-cmake-tools that might be able to make use of them. I created tests for these changes as well. All the other 680 tests pass with these changes in place.

I tried to upload a diff between my local branch and the master commit hash 69949719c8b0c419ef435e28c02507cc0a4806f4. But I am too new. So I can’t upload. I’ll spare you pasting it into this topic unless there is interest.

Does adding PROGRAM and ARGS fields to add_test() seem like a reasonable feature to be added?

Thanks
Chris

2

How would a call to add_test() look with your proposed PROGRAM and ARGS keywords? Would those replace the COMMAND, or still require that to be added as well? At the moment, I can’t see how using PROGRAM and ARGS is any different to what COMMAND already provides, but maybe I’m just not reading your explanation carefully enough for why the first argument after COMMAND isn’t suitable.

Note that you should use the CROSSCOMPILING_EMULATOR or TEST_LAUNCHER target properties to specify a script to run the test somewhere else or on a host machine that can’t directly run your target binaries. Don’t list such a script in the COMMAND arguments in the call to add_test(). If you’re trying to do that, look at using one of these two test properties to solve your problem instead. There are global CMAKE_CROSSCOMPILING_EMULATOR and CMAKE_TEST_LAUNCHER variables you can use to specify them for the whole project.

3

I’ll take a look at those options. All I could find when I searched the web was to make the first element of command a script to set things up.

When I searched the web on how to run a test remotely using this:

cmake how to add a test that runs on a remote host

the AI overview said this:

AI Overview
CMake's CTest does not inherently provide a direct "remote run" capability for tests. However, you can achieve remote test execution by combining CTest with custom commands or scripts that handle the remote interaction.
Steps to add a test that runs on a remote host using CMake and CTest:
enable testing.
In your top-level CMakeLists.txt file, enable testing using enable_testing():
Code

    enable_testing()

Define the Remote Execution Command.
Instead of directly executing your test binary, you will define a command that executes the test on the remote host. This command will typically involve an SSH client or a similar remote execution tool. Using ssh.
You can use ssh to execute commands on the remote machine. For example:
Code

        add_test(
            NAME MyRemoteTest
            COMMAND ssh user@remote_host "/path/to/remote/test_executable arg1 arg2"
            WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
        )

Replace user@remote_host with your remote credentials and /path/to/remote/test_executable with the actual path to your compiled test executable on the remote machine. Using a Wrapper Script.

So, with that recommendation I defined:

add_test(
  NAME UnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*)

Notice first entry of COMMAND is a shell script.

#!/bin/bash
REMOTE_HOST="chrisk@qwtest.local"
REMOTE_PATH="/usr/local/qw/tests"
LOCAL_BINARY="$1" # TestExecutable
shift 
# Transfer the binary
scp "${LOCAL_BINARY}" "${REMOTE_HOST}:${REMOTE_PATH}/"
# Execute remotely and capture result
ssh "${REMOTE_HOST}" "cd ${REMOTE_PATH} && ./${LOCAL_BINARY} $@"
exit $?

I run the test and it works fine. The script copies the executable over to the remote host and executes it and it all passes when just running the command.

If I try to debug this it fails. In debug mode vs-cmake-tool uses the “program” value to find the local file that has the symbol table to pass to gdb. The vs-cmake-tool defines testProgram, which is recommended to define the “program” field, as the first element of COMMAND. Of course this is a shell script so when gdb tries to look for the symbol table it fails with wrong file format.

If I change the COMMAND in the test to:

add_test(
  NAME UnitsTemperatureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*)

Here I define COMMAND so that the binary for the remote host archtiecture is the first element so it gets defined to be testProgram. Now if I run this test in debug mode it all works fine. The first element of COMMAND is the binary that has the symbol table. This makes gdb happy when it gets fired off. The debugger comes up and stops at the first break point on the remote target.

My solution right now is to have two add_test’s:

add_test(
  NAME DebugUnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*
  )

add_test(
  NAME UnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh
    units_pressure_test
    --gtest_output=json:units_pressure_test_results.json
    --gtest_filter=UnitsPressureTest.*
  )

On the vscode Test explorer it shows up as:

DebugUnitsPressureTest
UnitsPressureTest

I have to remember to select UnitsPressureTest when I just want to run the test. I have to select DebugUnitsPressureTest if I want to debug the test.

This is what I am trying to solve. How do I get just one add_test() that allows me to run a test remotely and also allows me to debug the test remotely.

I will check into CROSSCOMPILING_EMULATOR and TEST_LAUNCHER to see if that allows me to do this. That would be much better than modifying any code.

Thanks