add_test() assumes test executable is first string after COMMAND

Code
,
1

I am using cmake through vscode to cross-compile from a Ubuntu host to a Raspberry Pi Zero 2W target. I need to run the tests on the remote host. I created the following add_test:

add_test(
  NAME UnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*)

Since I need to run the test on a remote host the COMMAND is ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh and it simply copies the test executable to the remote and then runs it on the remote:

#!/bin/bash
REMOTE_HOST="chrisk@qwtest.local"
REMOTE_PATH="/usr/local/qw/tests"
LOCAL_BINARY="$1" # TestExecutable
shift 
# Transfer the binary
scp "${LOCAL_BINARY}" "${REMOTE_HOST}:${REMOTE_PATH}/"
# Execute remotely and capture result
ssh "${REMOTE_HOST}" "cd ${REMOTE_PATH} && ./${LOCAL_BINARY} $@"
exit $?

When I run a test it works fine. The script is called and the test gets copied to the remote target and it gets executed. Everything passes and all is well.
In order to debug the test I need to create a configuration in launch.json that will copy the test to the remote host and start the gdbserver on the remote host. The launcher also needs to know where the test executable is in order to load the symbol table. The program field in the launch.json configuration defines this.
In vscode I have a launch.json configuration to debug the tests as below:

{
            "name": "Debug Tests",
            "cwd": "${cmake.testWorkingDirectory}",
            "request": "launch",
            "program": "${cmake.testProgram}",
            "args": ["${cmake.testArguments}"],
            "type": "cppdbg",
            "MIMode": "gdb",
            "miDebuggerPath": "/usr/bin/gdb-multiarch",
            "miDebuggerArgs": "",
            "miDebuggerServerAddress": "192.168.50.6:4711",
            "targetArchitecture": "arm64",
            "stopAtEntry": false,
            "preLaunchTask": "StartUnitTestGDBServer",
            "postDebugTask": "StopUnitTestGDBServer"
        }

The ${cmake.testProgram} is the suggested value in order to allow this one configuration to work for many different tests. When I try to debug the test from vscode’s Test explorer I get the following error:

GDB failed with message: "/home/chrisk/Projects/RaspberryPi/WS/cmake/run_remote_test.sh":not in executable format:file format not recognized.

Note that ${cmake.testProgram} resolves to run_remote_test.sh. This is the first string in the add_test() COMMAND property. It is the script I have to run to copy the executable over and run on the remote host when running the test.
I really want that value to be the second string units_pressure_test. That is the real test executable. But I can’t call it as the first string of the COMMAND because it needs to run on the remote host, not locally.

I have not been able to find a way to specify the program field in launch.json in a generic way that will work for all the tests.

Maybe there should be a way of separating the executable from the COMMAND field so they can be defined independently. Maybe add an EXECUTABLE option to add_test that overrides the rule of using the first string of COMMAND:

add_test(
  NAME UnitsPressureTests
  EXECUTABLE {CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*)

The EXECUTABLE value can define a ${cmake.testExecutable} that can be used in place of ${cmake.testProgram} as the launch.json program field when the first string after COMMAND is not the correct thing to use.

To verify things I made the units_pressure_test the first string after COMMAND:

add_test(
  NAME UnitsTemperatureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  #COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh units_pressure_test
  COMMAND ${CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*)

I was then able to run the debugger and it correctly copied the file to the remote host, started up gdbserver on the rtemote host and ran gdb to connect to it just fine. i was able to debug just fine. But, of course I can’t actually run the test because it will try to execute units_pressure_test locally which will fail because it is for the wrong architecture.

Here are all the software versions I am using
I am using cmake version:

$ cmake --version
cmake version 3.28.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

I am using it within Vscode. That about output shows:

Version: 1.103.1
Commit: 360a4e4fd251bfce169a4ddf857c7d25d1ad40da
Date: 2025-08-12T16:25:40.542Z
Electron: 37.2.3
ElectronBuildId: 12035395
Chromium: 138.0.7204.100
Node.js: 22.17.0
V8: 13.8.500258-electron.0
OS: Linux x64 6.8.0-71-generic snap

I am running on a Linux host with the following os-release output

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
2

This sounds like a vscode problem, not a CMake problem.

3

I asked VSCode and they said it was an extension problem. So, I will try to track down that extension. But, it still seems like there needs to be some extra field in add_test to specifically identify the actual test executable instead of the extension having to assume that it will always be the first element of the COMMAND option.

I’ll try to identify the extension and see where that leads.

4

It seems the extension is CMake Tools. I looked at the source. I don’t have much experience with this language but it seems in src/ctest.ts the add_start values get assigned here

// Commands can't be used to replace array (i.e., args); and both test program and test args requires folder and
        // test name as parameters, which means one lauch config for each test. So replacing them here is a better way.
        chosenConfig.config = this.replaceAllInObject<vscode.DebugConfiguration>(chosenConfig.config, '${cmake.testProgram}', this.testProgram(testName));
        chosenConfig.config = this.replaceAllInObject<vscode.DebugConfiguration>(chosenConfig.config, '${cmake.testWorkingDirectory}', this.testWorkingDirectory(testName));

        // Replace cmake.testArgs wrapped in quotes, like `"${command:cmake.testArgs}"`, without any spaces in between,
        // since we need to repalce the quotes as well.
        chosenConfig.config = this.replaceArrayItems(chosenConfig.config, '${cmake.testArgs}', this.testArgs(testName)) as vscode.DebugConfiguration;

the this.testProgram() function is defined in the same file:

private testProgram(testName: string): string {
        if (this.tests) {
            for (const test of this.tests.tests) {
                if (test.name === testName) {
                    return test.command[0];
                }
            }
        } else if (this.legacyTests) {
            for (const test of this.legacyTests) {
                if (test.name === testName) {
                    return test.name;
                }
            }
        }
        return '';
    }

I would expect that test.command[0] refers to the first string after the COMMAND in add_start(). With the data available in add_test() that seems like a good choice. However, if the test is compiled for a target of a different architecture it is not a good choice since that test can’t be run on the build host. But with the current info available in add_test() it is the best choice.

It looks like if it is a Legacy the name of the test became the testProgram value. But, it seems wrong to go backwards and setting legacy may do other unwanted things.

It seems reasonable to add a new field defining a new property that contains the executable and have a new testExecutable or something similar (testDebugProgram??). Some way to provide a value that is not dependent on the COMMAND used to run the tests.

Also, the testArgs will still be wrong as they are based on the position in test.command array.

private testArgs(testName: string): string[] {
        if (this.tests) {
            for (const test of this.tests.tests) {
                if (test.name === testName) {
                    return test.command.slice(1);
                }
            }
        }
        return [];
    }

I am not real familiar with this language, but I assume test.command.slice(1) means all elements in test.command except offset 0.

So, potentially a separate field could be considered for the arguments so that you can define some arguments independent of the COMMAND property.

It seems that CMake needs some extra properties in add_test() and then the extension needs to define additional variables to make them available in VScode.

5

Du you know CMAKE_CROSSCOMPILING_EMULATOR?

Let’s say you want to run your tests on a Raspberry Pi via SSH.

File: run_remote.sh

#!/usr/bin/env bash
set -e

TARGET_HOST="pi@192.168.1.42"
TARGET_DIR="/tmp/cmake_test"
REMOTE_BINARY="$TARGET_DIR/$(basename "$1")"

# 1. Copy binary to remote
ssh "$TARGET_HOST" "mkdir -p $TARGET_DIR"
scp "$1" "$TARGET_HOST:$REMOTE_BINARY"

# 2. Shift script args so remaining are test arguments
shift

# 3. Run binary on remote and forward stdout/stderr
ssh "$TARGET_HOST" "$REMOTE_BINARY" "$@"

Than you my use it like this:

set(CMAKE_CROSSCOMPILING_EMULATOR
    ${CMAKE_SOURCE_DIR}/run_remote.sh
    CACHE STRING "Command to run executables on target")

add_executable(mytest test.cpp)
add_test(NAME mytest COMMAND mytest)
6

I tried that and I couldn’ t get it to work when trying to just run a test rather than debugging it.

[ctest] Test project /home/chrisk/Projects/RaspberryPi/WS/build
[ctest]     Start 2: DebugUnitsPressureTests
[ctest] 1/1 Test #2: DebugUnitsPressureTests ..........***Failed    0.00 sec
[ctest] /home/chrisk/Projects/RaspberryPi/WS/build/src/lib/qw/units/pressure/tests/units_pressure_test: 1: Syntax error: word unexpected (expecting ")")
[ctest] 
[ctest] 
[ctest] 0% tests passed, 1 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.00 sec
[ctest] 
[ctest] The following tests FAILED:
[ctest] 	  2 - DebugUnitsPressureTests (Failed)
[ctest] Errors while running CTest

Essentially it tries to run the binary on with arch for remote machine on the local machine.
I do have CMAKE_SYSTEM_NAME set to Linux, That should enable CMAKE_CROSSCOMPILING so CMAKE_CROSSCOMPILING_EMULATOR should be getting called. But it doesn’t seem to when running a test.

I have:

set(CMAKE_CROSSCOMPILING_EMULATOR
    ${CMAKE_SOURCE_DIR}/cmake/run_remote.sh
    CACHE STRING "Command to run executables on target")

add_executable(units_pressure_test units_pressure_test.cpp)
add_test(
  NAME DebugUnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*
  )

where run_remote looks like this

#!/usr/bin/env bash
set -e

TARGET_HOST="chris@qwtest.local"
TARGET_DIR="/tmp/cmake_test"
REMOTE_BINARY="$TARGET_DIR/$(basename "$1")"

# 1. Copy binary to remote
ssh "$TARGET_HOST" "mkdir -p $TARGET_DIR"
scp "$1" "$TARGET_HOST:$REMOTE_BINARY"

# 2. Shift script args so remaining are test arguments
shift

# 3. Run binary on remote and forward stdout/stderr
ssh "$TARGET_HOST" "$REMOTE_BINARY" "$@"

exit 0

For the moment I just have two add_tests. One I only run the test for debugging and the other to run the test without debugging.

add_test(
  NAME DebugUnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*
  )

add_test(
  NAME UnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh
    units_pressure_test
    --gtest_output=json:units_pressure_test_results.json
    --gtest_filter=UnitsPressureTest.*
  )

It’s a workable solution for the moment.

Thanks