Uploading images to CDash: ATTACHED_FILES vs. CTestMeasurement

Usage
1

TL;DR: 2 issues and 1 question for uploading files to CDash:

  • Images uploaded via ATTACHED_FILES are not shown directly but are presented as .tgz
  • Images uploaded via CTestMeasurement... are truncated to 64kb in a default database setup (MariaDB 10.6).
  • Why are there two different locations for storing uploaded images/files?

More Details:

I experimented with uploading images from tests with ctest, and found it nice that this already exists and is quite well supported! I did have minor issues though:

  • As reported in this discussion, images uploaded via the ATTACHED_FILES mechanism end up as .tgz files instead of being directly shown. It would be nice if CDash directly would display such image files (same as for files uploaded via the CTestMeasurement mechanism, see below). The tarball adds unnecessary additional steps for each check (in addition to opening the specific test, one needs to download, extract gz, etxtract tar, and open the image).

  • A workaround is to use the <CTestMeasurement ... test output as described here in the CTest documentation under Attached Files / Additional Test Measurements. Images uploaded with type="image/png" (or jpg/…) are shown directly in the test results - exactly what I want.
    This is however a bit more complex for some cases than simply setting the ATTACHED_FILES property. We do kind of integration tests, and so our tests directly run a (shipped) command line tool where we do not want to include logic for creating the <CTestMeasurement... XML output. So we would need to wrap the call to this tool into a custom script that runs the actual test, and afterwards outputs that required xml snippet.
    When I tested the <CTestMeasurement ... approach, I found that the uploaded image was truncated (CDash 3.4.0, database not completely newly created but probably not older than from 3.3.0). After some digging I found out that in the database in the image table, only the first 64KB were stored due to the img column being of type BLOB which can only story 64kb in mysql/mariadb. Is this intentional? While I agree that some restriction on the file sizes of uploaded test files does make sense, I did not see any mention of the 64kb limit in the documentation. After changing the column type to MEDIUMBLOB, everything worked as expected. Looking at the database setup script for the image table, this column uses the ‘binary’ type from Illuminate; the Illuminate documentation on binary says this maps to BLOB but does not explicitly mention whether there is a way to specify a “larger” blob type like mediumblob or longblob there - possibly by adding a size parameter to the binary call?

  • What I also found out is that files uploaded via ATTACHED_FILES are stored completely differently (these go into the testmeasurement table) than images uploaded via the CTestMeasurement mechanism (these go into the images table). The value column in the testmeasurement table uses a mediumtext (extended by this change, so it isn’t affected by the 64kb restrictions, though it uses a little more memory due to it being I think base64 encoded).
    I find it confusing that images uploaded via the CTestMeasurement do not go into testmeasurement; why are there even two storage locations for the same kind of data?

2

Just in case someone comes across this thread searching for help with uploading images, or in case I am overthinking this and there is a suggestion for a simpler solution:

To run arbitrary commands with image output, I have now written a custom cmake script (for platform independence) that runs a given command and triggers the XML output required by CDash to display the image:

${CMAKE_SOURCE_DIR}/test/run_test_with_img_output.cmake.in:

set(TEST_COMMAND "@TEST_COMMAND@")
message("Running ${TEST_COMMAND}")
execute_process(COMMAND ${TEST_COMMAND} RESULT_VARIABLE CommandResult)
message("<CTestMeasurementFile type=\"image/png\" name=\"@TEST_NAME@\">@TEST_OUTIMG@</CTestMeasurementFile>")
message("Command returned: ${CommandResult}")
if (NOT ("${CommandResult}" EQUAL "0"))
	message(SEND_ERROR ${CommandResult})
else()
	message("Success!")
endif()

And using it looks like this:

set(TEST_OUTIMG ${CMAKE_BINARY_DIR}/Testing/Temporary/outputimage.png)
set(TEST_NAME "TestName")
set(TEST_COMMAND ${TestBinary} ...other parameters ... --imageoutput ${TEST_OUTIMG})

set(TEST_SCRIPT_NAME "${CMAKE_BINARY_DIR}/${TEST_NAME}.cmake")
configure_file("${CMAKE_SOURCE_DIR}/test/run_test_with_img_output.cmake.in" ${TEST_SCRIPT_NAME} @ONLY)
add_test(NAME ${TEST_NAME} COMMAND ${CMAKE_COMMAND} -P ${TEST_SCRIPT_NAME} )

My current solution as described above is unfortunately much more lengthy than a solution with ATTACHED_FILES, which would just take two lines of cmake code:

add_test(NAME ${TEST_NAME} COMMAND ${TEST_COMMAND})
set_tests_properties(${TEST_NAME} PROPERTIES ATTACHED_FILES ${TEST_OUTIMG})
3

In case anyone is also interested in this, I have added a feature request for better support for images uploaded via ATTACHED_FILES.