Does CTest always return exit code 0?

Code
1

Been struggling with an Azure build and it turns out that I think CTest is not reporting back when a compile fails? Here is some output from Azure:

[20/1636] Building CXX object complex/CMakeFiles/complex.dir/src/complex/Core/Preferences.cpp.o
ninja: build stopped: subcommand failed.
Command exited with the value: 1
MakeCommand:/opt/local/cmake-3.26.4-linux-x86_64/bin/cmake --build . --config "${CTEST_CONFIGURATION_TYPE}"
   1 Compiler errors
   2 Compiler warnings
Error(s) when building project
##[debug]Debug text 'True''=True
##[debug]Debug text LASTEXITCODE=0

Here are the commands that we run:

      ctest -V -C $(build_type) -D $(model_type)Build
      ctest -C $(build_type) -D $(model_type)Submit

We are trying to report the result up to my.cdash.org which is why we think we should be running both ctest commands?

Even though the compile failed, ctest still returned 0. I tested this locally in a zsh shell (we are using powershell on Azure) and the same thing happens. I took a look through the ctest arguments and there does not seem to be anything about passing back the error if the compile or testing fails.

Thoughts?

Thanks
Mike Jackson

2

Odd. I’m not all that familiar with ctest -D at this point. I think a CTest script gives a lot more control (e.g., submitting even if the build fails).

3

I would also recommend the ctest -S script. So much control over what happens.

4

Thank you for the tips. I was finally able to get things figured out. For anyone who is possibly struggling with this same sort of setup (Azure self hosted build bots reporting results to CDash).

I created 3 CTest scripts:
.azure/azure_ci_configure.cmake
.azure/azure_ci_build.cmake
.azure/azure_ci_test.cmake

Each of those files do what the title suggests. Each file needs a ctest_start() where the “build” and “test” scripts need the ctest_start(APPEND) so that everything shows up correctly on CDash.

Inside the azure-pipelines.yml file we invoke the scripts like this:

#==================================================================================================
# CDash Section:
#    In this section we are going to call `ctest` 3 times in order to separate out the output from
# each CDash stage: Configure,Update :: Build :: Testing
# This makes the output easier to parse through for the testing.
#==================================================================================================

  - powershell: |
      cd $(Build.BinariesDirectory)
      ctest -VV -DCTEST_SOURCE_DIR=$(dream3d_source_dir) -DCTEST_BINARY_DIRECTORY=$(Build.BinariesDirectory) -Ddashboard_model=Experimental -DCTEST_DASHBOARD_ROOT=$(Build.Repository.LocalPath) -DCMAKE_PRESET_NAME="$(preset_name)" -DCTEST_CONFIGURATION_TYPE=Release -S $(Build.Repository.LocalPath)/DREAM3DNX/.azure/azure_ci_configure.cmake
    displayName: CDash Update/Configure
  
  - powershell: |
      cd $(Build.BinariesDirectory)
      ctest -VV -DCTEST_SOURCE_DIR=$(dream3d_source_dir) -DCTEST_BINARY_DIRECTORY=$(Build.BinariesDirectory) -Ddashboard_model=Experimental -DCTEST_DASHBOARD_ROOT=$(Build.Repository.LocalPath) -DCMAKE_PRESET_NAME="$(preset_name)" -DCTEST_CONFIGURATION_TYPE=Release -S $(Build.Repository.LocalPath)/DREAM3DNX/.azure/azure_ci_build.cmake
    displayName: CDash Build

  - powershell: |
      cd $(Build.BinariesDirectory)
      ctest -VV -DCTEST_SOURCE_DIR=$(dream3d_source_dir) -DCTEST_BINARY_DIRECTORY=$(Build.BinariesDirectory) -Ddashboard_model=Experimental -DCTEST_DASHBOARD_ROOT=$(Build.Repository.LocalPath) -DCMAKE_PRESET_NAME="$(preset_name)" -DCTEST_CONFIGURATION_TYPE=Release -S $(Build.Repository.LocalPath)/DREAM3DNX/.azure/azure_ci_test.cmake
    displayName: CDash Test

the “-VV” is not completely necessary except to allow us to monitor the builds/testing. The scripts will bail out with message(FATAL_ERROR “”) if anything goes wrong which will force ctest to return a non-zero return code.

I think those are the major pieces of the puzzle, or at least how we decided to solve the problem

–
Mike Jackson