# CTest combines stdout and stderr, change to allow them to be treated differently

**URL:** https://discourse.cmake.org/t/ctest-combines-stdout-and-stderr-change-to-allow-them-to-be-treated-differently/9857
**Category:** Development
**Tags:** tool:ctest
**Created:** [January 15, 2024, 2:11pm UTC](https://discourse.cmake.org/t/ctest-combines-stdout-and-stderr-change-to-allow-them-to-be-treated-differently/9857 "2024-01-15T14:11:34Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![WilliamGLYork](https://discourse.cmake.org/user_avatar/discourse.cmake.org/williamglyork/32/4156_2.png) [@WilliamGLYork](https://discourse.cmake.org/u/WilliamGLYork)
#### Post date: [January 15, 2024, 2:11pm UTC](https://discourse.cmake.org/t/ctest-combines-stdout-and-stderr-change-to-allow-them-to-be-treated-differently/9857/1 "2024-01-15T14:11:34Z")

</div>

_cmake/Source/CTest/cmProcess.cxx_\*

```auto
//starts line 93
  uv_pipe_open(pipe_reader, fds[0]);
  uv_pipe_open(pipe_writer, fds[1]);
 
  uv_stdio_container_t stdio[3];
  stdio[0].flags = UV_INHERIT_FD;
  stdio[0].data.fd = 0;
  stdio[1].flags = UV_INHERIT_STREAM;
  stdio[1].data.stream = pipe_writer;
  stdio[2] = stdio[1];
 
  uv_process_options_t options = uv_process_options_t();
  options.file = this->Command.data();
  options.args = const_cast<char**>(this->ProcessArgs.data());
  options.stdio_count = 3; // in, out and err
  options.exit_cb = &cmProcess::OnExitCB;
  options.stdio = stdio;
 

```

**Problem**  
The above initialises all 3 file descriptors; “in out and err” but stderr is not used as it is just run through stdout `stdio[2] = stdio[1]` . I wish to use the outputof these two separate file descriptors separately.

**Changes**  
A new property for tests `add_test()` called `MERGE_OUTPUTS` should be introduced. Default set to true, which would keep the same behavior as current.  
But when set to false, `--output-log` and `--output-junit` would have seperate output fields for stdout and stderr.

**Conflicts**  
Regex properties, (ie PASS\_ FAIL\_ SKIP\_) would not be affected and run the same as before.  
I have been informed that this may affect DartMeasurement but this is not an area I have knowledge of so would appreciate any advice on this.

**Extension**  
This could also be introduced as a flag when running ctest `--merge-output` that would override the test’s properties.

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [January 16, 2024, 1:16pm UTC](https://discourse.cmake.org/t/ctest-combines-stdout-and-stderr-change-to-allow-them-to-be-treated-differently/9857/2 "2024-01-16T13:16:34Z")

</div>

> [@WilliamGLYork](#):
>
> Regex properties, (ie PASS\_ FAIL\_ SKIP\_) would not be affected and run the same as before.

Note that the regexes could have been matching on the combined output where they’d fail to match on the separated output. Also, with a split output, which output(s) do the expressions get matched against?

> [@WilliamGLYork](#):
>
> This could also be introduced as a flag when running ctest `--merge-output` that would override the test’s properties.

Test properties override global requests; the best this could do is change the default for `MERGE_OUTPUTS` when not otherwise specified…

---

<div class="post-metadata">

### Author: ![WilliamGLYork](https://discourse.cmake.org/user_avatar/discourse.cmake.org/williamglyork/32/4156_2.png) [@WilliamGLYork](https://discourse.cmake.org/u/WilliamGLYork)
#### Post date: [January 29, 2024, 4:19pm UTC](https://discourse.cmake.org/t/ctest-combines-stdout-and-stderr-change-to-allow-them-to-be-treated-differently/9857/3 "2024-01-29T16:19:26Z")

</div>

I am currently working on the assumption that I can include `MERGE_OUTPUTS` as a property of a test. Set by `set_tests_properties()`. This adds the value to the property map to be checked later, which fits with my intentions. However, this appears to store the properties as strings and my value is best stored as a boolean.

What would you advise me to do? Should I continue with this approach checking the value with `cmValue::IsOn`, or should I add as an “option” that is declared with `add_test()` similar to `COMMAND_EXPAND_LISTS`.

What fits best within the CMake guidelines/approach?

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.cmake.org/u/ben.boeckel)
#### Post date: [January 30, 2024, 3:40am UTC](https://discourse.cmake.org/t/ctest-combines-stdout-and-stderr-change-to-allow-them-to-be-treated-differently/9857/4 "2024-01-30T03:40:03Z")

</div>

`cmValue::IsOn` is the way to treat properties as booleans. Properties are better than arguments as well (especially when they are execution-related rather than “how to interpret/store the arguments to the `add_test` command” that `COMMAND_EXPAND_LISTS` is).

---

<div class="post-metadata">

### Author: ![scivision](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/s/a87d85/32.png) [@scivision](https://discourse.cmake.org/u/scivision)
#### Post date: [February 1, 2024, 8:11pm UTC](https://discourse.cmake.org/t/ctest-combines-stdout-and-stderr-change-to-allow-them-to-be-treated-differently/9857/5 "2024-02-01T20:11:58Z")

</div>

I would be strongly in support of option/property to separate stdout and stderr for CTest. I have tests/programs that output up to megabytes of text on stdout by design, but only emit necessary warnings/errors on stderr.

The ability to separate stdout and stderr with CTest would be a significant help in maintaining any tests with a lot of non-diagnostic text output on stdout.

---

<div class="post-metadata">

### Author: ![WilliamGLYork](https://discourse.cmake.org/user_avatar/discourse.cmake.org/williamglyork/32/4156_2.png) [@WilliamGLYork](https://discourse.cmake.org/u/WilliamGLYork)
#### Post date: [February 7, 2024, 4:33pm UTC](https://discourse.cmake.org/t/ctest-combines-stdout-and-stderr-change-to-allow-them-to-be-treated-differently/9857/6 "2024-02-07T16:33:22Z")

</div>

There are several cases in the below file that is used to handle the pipe streams that specify that the err stream is not to be used and instead it is merged into the out stream.

I have been trying to separate these two streams, but upon examining this file it seems to me that this may be a design feature rather than just out of simplicity. Do you know why this decision was made to combine these streams?

**cmUVProcessChain.cxx**

```auto
cmUVProcessChainBuilder& cmUVProcessChainBuilder::SetMergedBuiltinStreams()
{
  this->MergedBuiltinStreams = true;
  return this->SetBuiltinStream(Stream_OUTPUT).SetBuiltinStream(Stream_ERROR);
}

```

---

<div class="post-metadata">

### Author: ![brad.king](https://discourse.cmake.org/user_avatar/discourse.cmake.org/brad.king/32/11_2.png) [@brad.king](https://discourse.cmake.org/u/brad.king)
#### Post date: [February 7, 2024, 5:01pm UTC](https://discourse.cmake.org/t/ctest-combines-stdout-and-stderr-change-to-allow-them-to-be-treated-differently/9857/7 "2024-02-07T17:01:44Z")

</div>

All the cases where stdout/stderr are merged are intentional. The quoted `cmUVProcessChainBuilder` source is one implementation of that, but CTest’s test process management predates that abstraction and uses the code shown at the top of this topic.

The decision was made long ago in the early days of CMake/CTest because **merging stdout and stderr captures the relative order of messages on both pipes** (modulo buffer flushing). Everything a test prints to stdout or stderr is considered one block of output associated with the test. The `Test.xml` format used to send test output to CDash has only one field. It’s just always been that way and this is the first time I recall the possibility of splitting them has been raised.

For reference, `ninja` also does this with the output of individual build commands. The output of the build commands is considered normal operation for `ninja` regardless of whether the tools print it to stdout or stderr, so it all goes to `ninja -v`’s stdout. CTest’s `CTEST_USE_LAUNCHERS` feature wraps individual compiler invocations with a launcher that captures stdout and stderr separately before passing them through to the calling process.
