How I add a test that has multiple commands

Code
1

Sometimes you want to run multiple commands as part of a test. For instance, I wanted to:

  • Run my program with command-line arguments to generate an image
  • Run a program to compare the generated image with a reference image

I wanted my test to fail if either program fails, or if the images were different according to various metrics.

However, add_test only takes a single COMMAND. I solved this by having the command supplied to add_test be a CMake script that I supply with -D arguments to communicate variables from my CMakeLists.txt to the test case. I encapsulate this inside a function so that I can easily add tests.

My CMake script that runs the test (simplified):

list(APPEND PARAMETERS "savename=${TEST_SAVE_IMAGE}" "savedir=.")

file(REMOVE "${TEST_SAVE_IMAGE}")
execute_process(COMMAND "${ID}" ${PARAMETERS}
    COMMAND_ERROR_IS_FATAL ANY
    COMMAND_ECHO ${COMMAND_ECHO})
file(RENAME "${TEST_SAVE_IMAGE}" "${TEST_KEEP_IMAGE}/${TEST_SAVE_IMAGE}")

execute_process(COMMAND "${IMAGE_COMPARE}" "${GOLD_IMAGE}" "${TEST_KEEP_IMAGE}/${TEST_SAVE_IMAGE}"
    COMMAND_ERROR_IS_FATAL ANY
    COMMAND_ECHO ${COMMAND_ECHO})

My CMake helper function that adds the test case by running the script:

function(add_image_test name)
    cmake_parse_arguments(IMAGE_TEST "" "" "PARAMETERS" ${ARGN})
    list(APPEND IMAGE_TEST_PARAMETERS "batch=yes" "video=F6")
    add_test(NAME ImageTest.${name}
        COMMAND ${CMAKE_COMMAND}
            "-DID=$<SHELL_PATH:$<TARGET_FILE:id>>"
            "-DIMAGE_COMPARE=$<SHELL_PATH:$<TARGET_FILE:image-compare>>"
            "-DGOLD_IMAGE=${CMAKE_CURRENT_SOURCE_DIR}/gold-${name}.gif"
            "-DPARAMETERS=${IMAGE_TEST_PARAMETERS}"
            "-DTEST_SAVE_IMAGE=test-${name}.gif"
            "-DTEST_KEEP_IMAGE=${CMAKE_CURRENT_BINARY_DIR}"
            -P "${CMAKE_CURRENT_SOURCE_DIR}/image_compare_test.cmake"
        WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}/home"
    )
    set_tests_properties(ImageTest.${name} PROPERTIES LABELS "image")
endfunction()

An example of adding a test case:

add_image_test(fractint-formula
    PARAMETERS "type=formula" "formulafile=example2.frm" "formulaname=fractint")
2

This is the way, you need to run a script that executes your commands

3

If you need it to be one test, then putting it in a script is what I’d do too. However, you could also consider splitting this into two tests and use a test fixture to express the dependency between the two commands. Make the first test generate the image, and set that test’s FIXTURES_SETUP property. Make the command that compares images the second test, and set its FIXTURES_REQUIRED property.

An advantage of this arrangement is that you can easily run either step independently if you ever want to. For example, you might be working on just the part that compares images, so you don’t need to regenerate the same image each time. It will also take care of skipping the image compare step if there’s a problem generating the image.

1 Like
4

I have a similar setup except using regular tests with their DEPENDS properties. Then if I have many such tests I use a cmake function to generate the test cases and link them together.

1 Like