CTest combines stdout and stderr, change to allow them to be treated differently

Development
1

cmake/Source/CTest/cmProcess.cxx*

//starts line 93
  uv_pipe_open(pipe_reader, fds[0]);
  uv_pipe_open(pipe_writer, fds[1]);
 
  uv_stdio_container_t stdio[3];
  stdio[0].flags = UV_INHERIT_FD;
  stdio[0].data.fd = 0;
  stdio[1].flags = UV_INHERIT_STREAM;
  stdio[1].data.stream = pipe_writer;
  stdio[2] = stdio[1];
 
  uv_process_options_t options = uv_process_options_t();
  options.file = this->Command.data();
  options.args = const_cast<char**>(this->ProcessArgs.data());
  options.stdio_count = 3; // in, out and err
  options.exit_cb = &cmProcess::OnExitCB;
  options.stdio = stdio;

Problem
The above initialises all 3 file descriptors; “in out and err” but stderr is not used as it is just run through stdout stdio[2] = stdio[1] . I wish to use the outputof these two separate file descriptors separately.

Changes
A new property for tests add_test() called MERGE_OUTPUTS should be introduced. Default set to true, which would keep the same behavior as current.
But when set to false, --output-log and --output-junit would have seperate output fields for stdout and stderr.

Conflicts
Regex properties, (ie PASS_ FAIL_ SKIP_) would not be affected and run the same as before.
I have been informed that this may affect DartMeasurement but this is not an area I have knowledge of so would appreciate any advice on this.

Extension
This could also be introduced as a flag when running ctest --merge-output that would override the test’s properties.

1 Like
2

Note that the regexes could have been matching on the combined output where they’d fail to match on the separated output. Also, with a split output, which output(s) do the expressions get matched against?

Test properties override global requests; the best this could do is change the default for MERGE_OUTPUTS when not otherwise specified…

3

I am currently working on the assumption that I can include MERGE_OUTPUTS as a property of a test. Set by set_tests_properties(). This adds the value to the property map to be checked later, which fits with my intentions. However, this appears to store the properties as strings and my value is best stored as a boolean.

What would you advise me to do? Should I continue with this approach checking the value with cmValue::IsOn, or should I add as an “option” that is declared with add_test() similar to COMMAND_EXPAND_LISTS.

What fits best within the CMake guidelines/approach?

4

cmValue::IsOn is the way to treat properties as booleans. Properties are better than arguments as well (especially when they are execution-related rather than “how to interpret/store the arguments to the add_test command” that COMMAND_EXPAND_LISTS is).

5

I would be strongly in support of option/property to separate stdout and stderr for CTest. I have tests/programs that output up to megabytes of text on stdout by design, but only emit necessary warnings/errors on stderr.

The ability to separate stdout and stderr with CTest would be a significant help in maintaining any tests with a lot of non-diagnostic text output on stdout.

6

There are several cases in the below file that is used to handle the pipe streams that specify that the err stream is not to be used and instead it is merged into the out stream.

I have been trying to separate these two streams, but upon examining this file it seems to me that this may be a design feature rather than just out of simplicity. Do you know why this decision was made to combine these streams?

cmUVProcessChain.cxx

cmUVProcessChainBuilder& cmUVProcessChainBuilder::SetMergedBuiltinStreams()
{
  this->MergedBuiltinStreams = true;
  return this->SetBuiltinStream(Stream_OUTPUT).SetBuiltinStream(Stream_ERROR);
}
7

All the cases where stdout/stderr are merged are intentional. The quoted cmUVProcessChainBuilder source is one implementation of that, but CTest’s test process management predates that abstraction and uses the code shown at the top of this topic.

The decision was made long ago in the early days of CMake/CTest because merging stdout and stderr captures the relative order of messages on both pipes (modulo buffer flushing). Everything a test prints to stdout or stderr is considered one block of output associated with the test. The Test.xml format used to send test output to CDash has only one field. It’s just always been that way and this is the first time I recall the possibility of splitting them has been raised.

For reference, ninja also does this with the output of individual build commands. The output of the build commands is considered normal operation for ninja regardless of whether the tools print it to stdout or stderr, so it all goes to ninja -v’s stdout. CTest’s CTEST_USE_LAUNCHERS feature wraps individual compiler invocations with a launcher that captures stdout and stderr separately before passing them through to the calling process.