# Does CTest always return exit code 0?

**URL:** <https://discourse.cmake.org/t/does-ctest-always-return-exit-code-0/9257>\
**Category:** Code\
**Created:** [October 23, 2023, 4:42pm UTC](https://discourse.cmake.org/t/does-ctest-always-return-exit-code-0/9257 "2023-10-23T16:42:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![imikejackson](https://discourse.cmake.org/user_avatar/discourse.cmake.org/imikejackson/32/301_2.png) [@imikejackson](https://discourse.cmake.org/u/imikejackson)\
**Post date:** [October 23, 2023, 4:42pm UTC](https://discourse.cmake.org/t/does-ctest-always-return-exit-code-0/9257/1 "2023-10-23T16:42:03Z")

</div>

Been struggling with an Azure build and it turns out that I think CTest is not reporting back when a compile fails? Here is some output from Azure:

```auto
[20/1636] Building CXX object complex/CMakeFiles/complex.dir/src/complex/Core/Preferences.cpp.o
ninja: build stopped: subcommand failed.
Command exited with the value: 1
MakeCommand:/opt/local/cmake-3.26.4-linux-x86_64/bin/cmake --build . --config "${CTEST_CONFIGURATION_TYPE}"
   1 Compiler errors
   2 Compiler warnings
Error(s) when building project
##[debug]Debug text 'True''=True
##[debug]Debug text LASTEXITCODE=0

```

Here are the commands that we run:

```auto
      ctest -V -C $(build_type) -D $(model_type)Build
      ctest -C $(build_type) -D $(model_type)Submit

```

We are trying to report the result up to [my.cdash.org](http://my.cdash.org) which is why we _think_ we should be running both `ctest` commands?

Even though the compile failed, ctest still returned 0. I tested this locally in a zsh shell (we are using powershell on Azure) and the same thing happens. I took a look through the ctest arguments and there does not seem to be anything about passing back the error if the compile or testing fails.

Thoughts?

Thanks  
Mike Jackson

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)\
**Post date:** [October 23, 2023, 6:57pm UTC](https://discourse.cmake.org/t/does-ctest-always-return-exit-code-0/9257/2 "2023-10-23T18:57:59Z")

</div>

Odd. I’m not all that familiar with `ctest -D` at this point. I think a CTest script gives a lot more control (e.g., submitting even if the build fails).

---

<div class="post-metadata">

**Author:** ![scivision](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/s/a87d85/32.png) [@scivision](https://discourse.cmake.org/u/scivision)\
**Post date:** [October 23, 2023, 7:49pm UTC](https://discourse.cmake.org/t/does-ctest-always-return-exit-code-0/9257/3 "2023-10-23T19:49:39Z")

</div>

I would also recommend the `ctest -S` script. So much control over what happens.

---

<div class="post-metadata">

**Author:** ![imikejackson](https://discourse.cmake.org/user_avatar/discourse.cmake.org/imikejackson/32/301_2.png) [@imikejackson](https://discourse.cmake.org/u/imikejackson)\
**Post date:** [October 24, 2023, 1:36pm UTC](https://discourse.cmake.org/t/does-ctest-always-return-exit-code-0/9257/4 "2023-10-24T13:36:31Z")

</div>

Thank you for the tips. I was finally able to get things figured out. For anyone who is possibly struggling with this same sort of setup (Azure self hosted build bots reporting results to CDash).

I created 3 CTest scripts:  
.azure/azure\_ci\_configure.cmake  
.azure/azure\_ci\_build.cmake  
.azure/azure\_ci\_test.cmake

Each of those files do what the title suggests. Each file needs a ctest\_start() where the “build” and “test” scripts need the ctest\_start(APPEND) so that everything shows up correctly on CDash.

Inside the azure-pipelines.yml file we invoke the scripts like this:

```auto
#==================================================================================================
# CDash Section:
# In this section we are going to call `ctest` 3 times in order to separate out the output from
# each CDash stage: Configure,Update :: Build :: Testing
# This makes the output easier to parse through for the testing.
#==================================================================================================

  - powershell: |
      cd $(Build.BinariesDirectory)
      ctest -VV -DCTEST_SOURCE_DIR=$(dream3d_source_dir) -DCTEST_BINARY_DIRECTORY=$(Build.BinariesDirectory) -Ddashboard_model=Experimental -DCTEST_DASHBOARD_ROOT=$(Build.Repository.LocalPath) -DCMAKE_PRESET_NAME="$(preset_name)" -DCTEST_CONFIGURATION_TYPE=Release -S $(Build.Repository.LocalPath)/DREAM3DNX/.azure/azure_ci_configure.cmake
    displayName: CDash Update/Configure
  
  - powershell: |
      cd $(Build.BinariesDirectory)
      ctest -VV -DCTEST_SOURCE_DIR=$(dream3d_source_dir) -DCTEST_BINARY_DIRECTORY=$(Build.BinariesDirectory) -Ddashboard_model=Experimental -DCTEST_DASHBOARD_ROOT=$(Build.Repository.LocalPath) -DCMAKE_PRESET_NAME="$(preset_name)" -DCTEST_CONFIGURATION_TYPE=Release -S $(Build.Repository.LocalPath)/DREAM3DNX/.azure/azure_ci_build.cmake
    displayName: CDash Build

  - powershell: |
      cd $(Build.BinariesDirectory)
      ctest -VV -DCTEST_SOURCE_DIR=$(dream3d_source_dir) -DCTEST_BINARY_DIRECTORY=$(Build.BinariesDirectory) -Ddashboard_model=Experimental -DCTEST_DASHBOARD_ROOT=$(Build.Repository.LocalPath) -DCMAKE_PRESET_NAME="$(preset_name)" -DCTEST_CONFIGURATION_TYPE=Release -S $(Build.Repository.LocalPath)/DREAM3DNX/.azure/azure_ci_test.cmake
    displayName: CDash Test

```

the “-VV” is not completely necessary except to allow us to monitor the builds/testing. The scripts will bail out with message(FATAL\_ERROR “”) if anything goes wrong which will force ctest to return a non-zero return code.

I think those are the major pieces of the puzzle, or at least how we decided to solve the problem

–  
Mike Jackson