# add\_test() assumes test executable is first string after COMMAND

**URL:** https://discourse.cmake.org/t/add-test-assumes-test-executable-is-first-string-after-command/15012
**Category:** Code
**Tags:** os:linux, tool:ctest
**Created:** [August 15, 2025, 5:03am UTC](https://discourse.cmake.org/t/add-test-assumes-test-executable-is-first-string-after-command/15012 "2025-08-15T05:03:20Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![chriskot870](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/c/53a042/32.png) [@chriskot870](https://discourse.cmake.org/u/chriskot870)
#### Post date: [August 15, 2025, 5:03am UTC](https://discourse.cmake.org/t/add-test-assumes-test-executable-is-first-string-after-command/15012/1 "2025-08-15T05:03:20Z")

</div>

I am using cmake through vscode to cross-compile from a Ubuntu host to a Raspberry Pi Zero 2W target. I need to run the tests on the remote host. I created the following add\_test:

```auto
add_test(
  NAME UnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*)

```

Since I need to run the test on a remote host the COMMAND is ${CMAKE\_SOURCE\_DIR}/cmake/run\_remote\_test.sh and it simply copies the test executable to the remote and then runs it on the remote:

```auto
#!/bin/bash
REMOTE_HOST="chrisk@qwtest.local"
REMOTE_PATH="/usr/local/qw/tests"
LOCAL_BINARY="$1" # TestExecutable
shift 
# Transfer the binary
scp "${LOCAL_BINARY}" "${REMOTE_HOST}:${REMOTE_PATH}/"
# Execute remotely and capture result
ssh "${REMOTE_HOST}" "cd ${REMOTE_PATH} && ./${LOCAL_BINARY} $@"
exit $?

```

When I run a test it works fine. The script is called and the test gets copied to the remote target and it gets executed. Everything passes and all is well.  
In order to debug the test I need to create a configuration in launch.json that will copy the test to the remote host and start the gdbserver on the remote host. The launcher also needs to know where the test executable is in order to load the symbol table. The program field in the launch.json configuration defines this.  
In vscode I have a launch.json configuration to debug the tests as below:

```auto
{
            "name": "Debug Tests",
            "cwd": "${cmake.testWorkingDirectory}",
            "request": "launch",
            "program": "${cmake.testProgram}",
            "args": ["${cmake.testArguments}"],
            "type": "cppdbg",
            "MIMode": "gdb",
            "miDebuggerPath": "/usr/bin/gdb-multiarch",
            "miDebuggerArgs": "",
            "miDebuggerServerAddress": "192.168.50.6:4711",
            "targetArchitecture": "arm64",
            "stopAtEntry": false,
            "preLaunchTask": "StartUnitTestGDBServer",
            "postDebugTask": "StopUnitTestGDBServer"
        }

```

The ${cmake.testProgram} is the suggested value in order to allow this one configuration to work for many different tests. When I try to debug the test from vscode’s Test explorer I get the following error:

```auto
GDB failed with message: "/home/chrisk/Projects/RaspberryPi/WS/cmake/run_remote_test.sh":not in executable format:file format not recognized.

```

Note that ${cmake.testProgram} resolves to run\_remote\_test.sh. This is the first string in the add\_test() COMMAND property. It is the script I have to run to copy the executable over and run on the remote host when running the test.  
I really want that value to be the second string units\_pressure\_test. That is the real test executable. But I can’t call it as the first string of the COMMAND because it needs to run on the remote host, not locally.

I have not been able to find a way to specify the program field in launch.json in a generic way that will work for all the tests.

Maybe there should be a way of separating the executable from the COMMAND field so they can be defined independently. Maybe add an EXECUTABLE option to add\_test that overrides the rule of using the first string of COMMAND:

```auto
add_test(
  NAME UnitsPressureTests
  EXECUTABLE {CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*)

```

The EXECUTABLE value can define a ${cmake.testExecutable} that can be used in place of ${cmake.testProgram} as the launch.json program field when the first string after COMMAND is not the correct thing to use.

To verify things I made the units\_pressure\_test the first string after COMMAND:

```auto
add_test(
  NAME UnitsTemperatureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  #COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh units_pressure_test
  COMMAND ${CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*)

```

I was then able to run the debugger and it correctly copied the file to the remote host, started up gdbserver on the rtemote host and ran gdb to connect to it just fine. i was able to debug just fine. But, of course I can’t actually run the test because it will try to execute units\_pressure\_test locally which will fail because it is for the wrong architecture.

Here are all the software versions I am using  
I am using cmake version:

```auto
$ cmake --version
cmake version 3.28.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

```

I am using it within Vscode. That about output shows:

```auto
Version: 1.103.1
Commit: 360a4e4fd251bfce169a4ddf857c7d25d1ad40da
Date: 2025-08-12T16:25:40.542Z
Electron: 37.2.3
ElectronBuildId: 12035395
Chromium: 138.0.7204.100
Node.js: 22.17.0
V8: 13.8.500258-electron.0
OS: Linux x64 6.8.0-71-generic snap

```

I am running on a Linux host with the following os-release output

```auto
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

```

---

<div class="post-metadata">

### Author: ![LegalizeAdulthood](https://discourse.cmake.org/user_avatar/discourse.cmake.org/legalizeadulthood/32/5282_2.png) [@LegalizeAdulthood](https://discourse.cmake.org/u/LegalizeAdulthood)
#### Post date: [August 15, 2025, 3:40pm UTC](https://discourse.cmake.org/t/add-test-assumes-test-executable-is-first-string-after-command/15012/2 "2025-08-15T15:40:20Z")

</div>

This sounds like a vscode problem, not a CMake problem.

---

<div class="post-metadata">

### Author: ![chriskot870](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/c/53a042/32.png) [@chriskot870](https://discourse.cmake.org/u/chriskot870)
#### Post date: [August 15, 2025, 4:14pm UTC](https://discourse.cmake.org/t/add-test-assumes-test-executable-is-first-string-after-command/15012/3 "2025-08-15T16:14:59Z")

</div>

I asked VSCode and they said it was an extension problem. So, I will try to track down that extension. But, it still seems like there needs to be some extra field in add\_test to specifically identify the actual test executable instead of the extension having to assume that it will always be the first element of the COMMAND option.

I’ll try to identify the extension and see where that leads.

---

<div class="post-metadata">

### Author: ![chriskot870](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/c/53a042/32.png) [@chriskot870](https://discourse.cmake.org/u/chriskot870)
#### Post date: [August 15, 2025, 5:34pm UTC](https://discourse.cmake.org/t/add-test-assumes-test-executable-is-first-string-after-command/15012/4 "2025-08-15T17:34:24Z")

</div>

It seems the extension is CMake Tools. I looked at the source. I don’t have much experience with this language but it seems in src/ctest.ts the add\_start values get assigned here

```auto
// Commands can't be used to replace array (i.e., args); and both test program and test args requires folder and
        // test name as parameters, which means one lauch config for each test. So replacing them here is a better way.
        chosenConfig.config = this.replaceAllInObject<vscode.DebugConfiguration>(chosenConfig.config, '${cmake.testProgram}', this.testProgram(testName));
        chosenConfig.config = this.replaceAllInObject<vscode.DebugConfiguration>(chosenConfig.config, '${cmake.testWorkingDirectory}', this.testWorkingDirectory(testName));

        // Replace cmake.testArgs wrapped in quotes, like `"${command:cmake.testArgs}"`, without any spaces in between,
        // since we need to repalce the quotes as well.
        chosenConfig.config = this.replaceArrayItems(chosenConfig.config, '${cmake.testArgs}', this.testArgs(testName)) as vscode.DebugConfiguration;

```

the this.testProgram() function is defined in the same file:

```auto
private testProgram(testName: string): string {
        if (this.tests) {
            for (const test of this.tests.tests) {
                if (test.name === testName) {
                    return test.command[0];
                }
            }
        } else if (this.legacyTests) {
            for (const test of this.legacyTests) {
                if (test.name === testName) {
                    return test.name;
                }
            }
        }
        return '';
    }

```

I would expect that test.command[0] refers to the first string after the COMMAND in add\_start(). With the data available in add\_test() that seems like a good choice. However, if the test is compiled for a target of a different architecture it is not a good choice since that test can’t be run on the build host. But with the current info available in add\_test() it is the best choice.

It looks like if it is a Legacy the name of the test became the testProgram value. But, it seems wrong to go backwards and setting legacy may do other unwanted things.

It seems reasonable to add a new field defining a new property that contains the executable and have a new testExecutable or something similar (testDebugProgram??). Some way to provide a value that is not dependent on the COMMAND used to run the tests.

Also, the testArgs will still be wrong as they are based on the position in test.command array.

```auto
private testArgs(testName: string): string[] {
        if (this.tests) {
            for (const test of this.tests.tests) {
                if (test.name === testName) {
                    return test.command.slice(1);
                }
            }
        }
        return [];
    }

```

I am not real familiar with this language, but I assume test.command.slice(1) means all elements in test.command except offset 0.

So, potentially a separate field could be considered for the arguments so that you can define some arguments independent of the COMMAND property.

It seems that CMake needs some extra properties in add\_test() and then the extension needs to define additional variables to make them available in VScode.

---

<div class="post-metadata">

### Author: ![ClausKlein](https://discourse.cmake.org/user_avatar/discourse.cmake.org/clausklein/32/352_2.png) [@ClausKlein](https://discourse.cmake.org/u/ClausKlein)
#### Post date: [August 15, 2025, 7:29pm UTC](https://discourse.cmake.org/t/add-test-assumes-test-executable-is-first-string-after-command/15012/5 "2025-08-15T19:29:18Z")

</div>

Du you know [CMAKE\_CROSSCOMPILING\_EMULATOR](https://cmake.org/cmake/help/latest/variable/CMAKE_CROSSCOMPILING_EMULATOR.html)?

Let’s say you want to run your tests on a Raspberry Pi via SSH.

**File:** `run_remote.sh`

```bash
#!/usr/bin/env bash
set -e

TARGET_HOST="pi@192.168.1.42"
TARGET_DIR="/tmp/cmake_test"
REMOTE_BINARY="$TARGET_DIR/$(basename "$1")"

# 1. Copy binary to remote
ssh "$TARGET_HOST" "mkdir -p $TARGET_DIR"
scp "$1" "$TARGET_HOST:$REMOTE_BINARY"

# 2. Shift script args so remaining are test arguments
shift

# 3. Run binary on remote and forward stdout/stderr
ssh "$TARGET_HOST" "$REMOTE_BINARY" "$@"

```

Than you my use it like this:

```cmake
set(CMAKE_CROSSCOMPILING_EMULATOR
    ${CMAKE_SOURCE_DIR}/run_remote.sh
    CACHE STRING "Command to run executables on target")

add_executable(mytest test.cpp)
add_test(NAME mytest COMMAND mytest)

```

---

<div class="post-metadata">

### Author: ![chriskot870](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/c/53a042/32.png) [@chriskot870](https://discourse.cmake.org/u/chriskot870)
#### Post date: [August 16, 2025, 10:43pm UTC](https://discourse.cmake.org/t/add-test-assumes-test-executable-is-first-string-after-command/15012/6 "2025-08-16T22:43:47Z")

</div>

I tried that and I couldn’ t get it to work when trying to just run a test rather than debugging it.

```auto
[ctest] Test project /home/chrisk/Projects/RaspberryPi/WS/build
[ctest] Start 2: DebugUnitsPressureTests
[ctest] 1/1 Test #2: DebugUnitsPressureTests ..........***Failed 0.00 sec
[ctest] /home/chrisk/Projects/RaspberryPi/WS/build/src/lib/qw/units/pressure/tests/units_pressure_test: 1: Syntax error: word unexpected (expecting ")")
[ctest] 
[ctest] 
[ctest] 0% tests passed, 1 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) = 0.00 sec
[ctest] 
[ctest] The following tests FAILED:
[ctest] 2 - DebugUnitsPressureTests (Failed)
[ctest] Errors while running CTest

```

Essentially it tries to run the binary on with arch for remote machine on the local machine.  
I do have CMAKE\_SYSTEM\_NAME set to Linux, That should enable CMAKE\_CROSSCOMPILING so CMAKE\_CROSSCOMPILING\_EMULATOR should be getting called. But it doesn’t seem to when running a test.

I have:

```auto
set(CMAKE_CROSSCOMPILING_EMULATOR
    ${CMAKE_SOURCE_DIR}/cmake/run_remote.sh
    CACHE STRING "Command to run executables on target")

add_executable(units_pressure_test units_pressure_test.cpp)
add_test(
  NAME DebugUnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*
  )

where run_remote looks like this

#!/usr/bin/env bash
set -e

TARGET_HOST="chris@qwtest.local"
TARGET_DIR="/tmp/cmake_test"
REMOTE_BINARY="$TARGET_DIR/$(basename "$1")"

# 1. Copy binary to remote
ssh "$TARGET_HOST" "mkdir -p $TARGET_DIR"
scp "$1" "$TARGET_HOST:$REMOTE_BINARY"

# 2. Shift script args so remaining are test arguments
shift

# 3. Run binary on remote and forward stdout/stderr
ssh "$TARGET_HOST" "$REMOTE_BINARY" "$@"

exit 0

```

For the moment I just have two add\_tests. One I only run the test for debugging and the other to run the test without debugging.

```auto
add_test(
  NAME DebugUnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_CURRENT_BINARY_DIR}/units_pressure_test
  --gtest_output=json:units_pressure_test_results.json
  --gtest_filter=UnitsPressureTest.*
  )

add_test(
  NAME UnitsPressureTests
  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
  COMMAND ${CMAKE_SOURCE_DIR}/cmake/run_remote_test.sh
    units_pressure_test
    --gtest_output=json:units_pressure_test_results.json
    --gtest_filter=UnitsPressureTest.*
  )

```

It’s a workable solution for the moment.

Thanks