Sending more than pass/fail data to CDash - what is the data format CDash expects?

Code
1

At my previous employer, we used CDash to display statistics about the tests that were run. When the code was run in “test” mode, additional statistics were recorded about the code and saved to an XML file. These statistics were then available in CDash to plot as history files. Unfortunately I didn’t set the CDash up - I only ever used it. I’m now attempting to set up CTest with CDash with my new employer.

My question is : what is the data format the CDash is expecting such that it can display all this data?
The data is going to be something like:
testname: simpletest
statistic1: “numberOfSteps”
staticistic type: int
statistic value: 10
statistic2: “maximumQuality”
staticistic type: float
statistic value: 0.1
etc.

I know that one could compare these statistics from one test to the next run using a baseline set of test-data. Again, never wrote this, only know that it happened.
Anyone have any ideas how one might go about this? If this requires some scripting to occur post-testing, then fine.

2

Is the MEASUREMENT test property what you are looking for? Note that it is a static value determined at configure time, not a value computed as part of running the test.

If you want to have the test compute a value to be reported to CDash, there’s an undocumented capability for that. It’s been on my todo list to document it for longer than I want to admit, but you can find a starting point for basic details and links here.

1 Like
3

These are going to be values calculated from the code during the test. We have instrumented our code to record statistics about what it is doing. At the end of a run if a certain environment variable is set the code exports the statistics to a JSON format file. Our team may add or delete these statistics from the code but we don’t want to edit the cmake setup.

4

I think you documentation problem is best solved by an example. What I’ve found so far in dealing with CMake/CTest is the documentation is rather like the linux man pages. Great if you are looking for some fine detail about a command you already know the existence of but not so good if you don’t know what the command is or even how a thing may be performed. I’ve been dealing with CMake on and off for around 6 years and I still feel like a novice. Novices like examples to follow and edit.

5

Thanks for the breadcrumbs; here’s what I found:

VTK’s tests write text to stdout containing an XML fragment:

<DartMeasurement name="WallTime" type="numeric/double">42</DartMeasurement>

CTest scans the output for those XML fragments, transforms them, and outputs them in the Testing/**/Test.xml document in this format:

<NamedMeasurement name="WallTime" type="numeric/double">
    <Value>42</Value>
</NamedMeasurement>
6

This is how I’ve now got this to work. Assume I have 4 pieces of additional data I want stored in CDash:

  • numberSides - integer , 6
  • areaSides - double , 6.0
  • wasItASquare - boolean, true
  • nameOfTheResult - string , “TheUltimateAnswer”

Your program must write to std::out.
The output for my 4 pieces of data would look like this sent to std::out:

<DartMeasurement name="numberSides" type=numeric/integer">6</DartMeasurement>
<DartMeasurement name="areaSides" type=numeric/double">6.0</DartMeasurement>
<DartMeasurement name="wasItASquare" type=bool>"true"</DartMeasurement>
<DartMeasurement name="nameOfTheResult" type=string>"TheUltimateAnswer"</DartMeasurement>

CTest does all the hard work and packages these extra data sets up and sends to CDash. The data can be viewed and plotted in CDash.

7

@jverdicchio I edited your post to fix the formatting. Feel free to take a look, but it’s basically the usual Markdown formatting.

8

Craig, thank you very much.