Help converting most of a CI matrix to presets?

Usage
1

I’m trying to convert this build matrix to CMake presets.

My initial experiment has awful repetition, as each configure preset needs a matching build preset and a test preset.

To test it, I’m running it like this:

$CMAKE_DIR/cmake -S `pwd` --preset=$PRESET
$CMAKE_DIR/cmake --build  --preset=$PRESET
$CMAKE_DIR/ctest          --preset=$PRESET

To remove the need for all the repetition of presets, I considered doing this instead:

cmake -S `pwd` --preset=$PRESET
cmake --build  --preset=ci-build
ctest          --preset=ci-test

But how will this know which build directory to run in? The problem with that is that some later presets will have a different binaryDir.

Depending on the generator, and whether Unity builds are on, most of them will do this:

"binaryDir": "${sourceDir}/cmake-build-spaces/${presetName}"

And some will do this:

"binaryDir": "${sourceParentDir}/cmake-build-spaces/${sourceDirName}/${presetName}"

I’d appreciate any advice on any better ways of using CMake presets on CI, to simplify matrices…

4 Likes
2

Cc: @kyle.edwards

3

If there was a cross-platform way in a script to obtain the binary dir for a given preset, then I could have the CI job cd to that directory and build and test inside there, as an interim measure.

4

It looks like you’re using inheritance in your presets, which I was going to suggest for de-duplicating as much of the code as possible. I think what you’ve come up with is the most ideal for your use case.

You can parse the JSON file yourself to get this information. The format is cross-platform and is intended to be read by other IDEs and tools.

5

Thanks for the reply. I think a summary is that build and test presets do not yet give the advertised benefits for Continuous Integration builds…

The problem with my current solution will be the need to create 10 or more buildConfigs and the same of testConfigs… I’ll have to script the updating of the file, as doing that by hand on lots of repos will be too error-prone and time-consuming…

My concern with parsing the JSON file myself is that I’ll end up having to write code to handle the variables that CMake supported - which is more scripting that needs to be written and tested, and replicated between repos, and in Continuous Integration systems…

@craig.scott I have read your description of Presets in the 9th edition of your book - I wonder whether the above usage is something you might be to address in future editions… It seems like the test configs, for example, are more focussed around “I want to easily run n different subsets of tests for my one build configuration” - rather than “I want to easily run all the tests on each of my n different configure presets”.

6

I wonder if it might make sense to add a new cmake --resolve-preset XXX command (or some other equivalent) that allows you to print the final fully expanded content for a specified preset? It could produce just a single JSON object, which would then be much easier for user scripts, etc. to post-process since they wouldn’t have to implement all the macro expansion and so on.

1 Like
7

Or perhaps a cmake --resolved-presets out_file_name which produces a JSON file containing all non-hidden presets in fully expanded form?

8

Hi Craig,

Thanks very much for both the suggestions.

My concern with having any JSON file or similar to parse is that we then up back in the world of having to write scripts to run CMake again - for different platforms and scenarios… My feeling is that needing to do anything with JSON output files to run builds and tests makes presets pointless, which is really sad.

I’ve spent today considering lots of different ideas, and the only one that I think will fly is to have the ability to pass a configPreset in as an extra argument to build and test…

Something like:

$CMAKE_DIR/cmake -S `pwd` --preset=$PRESET
$CMAKE_DIR/cmake --build  --config-preset=$PRESET --preset=ci-build
$CMAKE_DIR/ctest          --config-preset=$PRESET --preset=ci-test

This allows the full lifecycle to be run from the source directory, without needing to cd to the build location, and back again afterwards… Which is really nice for running via console too…

4 Likes
9

Just to add to the above…

By adding an extra argument to “cmake --build” and “ctest” that tells them which configure preset to use, then cmake can obtain the build directory in its usual way - and use that to cd to the build space…

This saves users from:

  • Having to script the build location
  • Having to create a load of boiler-plate duplicated build and test configuration, solely to specify the build location
3 Likes
10

Hi, I want to +1 Clare’s suggested solution. I’ve run into the same limitations, and have received similar feedback (on the need to maintain a large matrix of Configure Presets x Build Presets x Test Presets) from customers using the integration in VS and VS Code.

Even outside of a CI pipeline, this solution would remove the need to maintain nearly identical Build Presets and Test Presets for each Configure Preset in your matrix. Even with inheritance I’m finding that there is a lot of boilerplate to maintain.

From the command line, I’ll sometimes configure with a Configure Preset and then cd into the build directory and invoke CMake directly (without a Build Preset) to avoid maintaining a matrix of Build Presets. However, this means that any environment variables I’ve set in the Configure Preset won’t flow through to the build step. Clare’s solution would also help address this problem.

@kyle.edwards (or others) - do you have any concerns with Clare’s proposed solution?

2 Likes
11

Quick update - this solution would also address buildPresets - configurePresets one to one mapping?

2 Likes
12

Here’s another request to avoid having to parse the presets to find the build directory - this time, with regard to installation: Install components with presets

13

All of the proposed solutions would help me.

  • specifiing both configure and test/build preset is highly valuable for code (config) deduplication.
  • Resolving presets with CMake will enable more custom stuff like installing components.
14

I’m revisiting this problem. Are there any recent CMake enhancements that improve on the need to have a one-to-one mapping of configure and build preset?

15

The coming CMake 3.23 will allow includes:
https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html?highlight=presets#includes

16

@trmagma Thanks for your reply but I don’t understand how ‘includes’ help with this problem.

1 Like
17

For awareness, there is now a feature request in the CMake issue tracker for the suggestion quoted above: https://gitlab.kitware.com/cmake/cmake/-/issues/24827

1 Like
18

IMHO, it would a little help to use workflow presets like this?
Too, there is much repetition to write in this CMakePresets.json file!

{
  "version": 6,
  "cmakeMinimumRequired": {
    "major": 3,
    "minor": 25,
    "patch": 0
  },
  "configurePresets": [
    {
      "name": "ci-config-base",
      "hidden": true,
      "binaryDir": "${sourceDir}/cmake-build-spaces/${presetName}"
    },
    {
      "name": "ci-compiler-clang",
      "hidden": true,
      "cacheVariables": {
        "CMAKE_C_COMPILER": "clang-16",
        "CMAKE_CXX_COMPILER": "clang++-16"
      }
    },
    {
      "name": "ci-compiler-gcc",
      "hidden": true,
      "cacheVariables": {
        "CMAKE_C_COMPILER": "gcc-12",
        "CMAKE_CXX_COMPILER": "g++-12"
      }
    },
    {
      "name": "ci-generator-ninja",
      "hidden": true,
      "generator": "Ninja"
    },
    {
      "name": "ci-clang",
      "inherits": [
        "ci-config-base",
        "ci-compiler-clang",
        "ci-generator-ninja"
      ]
    },
    {
      "name": "ci-gcc",
      "inherits": [
        "ci-config-base",
        "ci-compiler-gcc",
        "ci-generator-ninja"
      ]
    }
  ],
  "buildPresets": [
    {
      "name": "ci-build-base",
      "configurePreset": "ci-config-base",
      "hidden": true,
      "jobs": 4
    },
    {
      "name": "ci-clang",
      "inherits": "ci-build-base",
      "configurePreset": "ci-clang"
    },
    {
      "name": "ci-gcc",
      "inherits": "ci-build-base",
      "configurePreset": "ci-gcc"
    }
  ],
  "testPresets": [
    {
      "name": "ci-test-base",
      "configurePreset": "ci-config-base",
      "hidden": true,
      "output": {
        "outputOnFailure": true
      }
    },
    {
      "name": "ci-clang",
      "inherits": "ci-test-base",
      "configurePreset": "ci-clang"
    },
    {
      "name": "ci-gcc",
      "inherits": "ci-test-base",
      "configurePreset": "ci-gcc"
    }
  ],
  "packagePresets": [
    {
      "name": "ci-clang",
      "configurePreset": "ci-clang",
      "generators": [
        "TGZ"
      ]
    },
    {
      "name": "ci-gcc",
      "configurePreset": "ci-gcc",
      "generators": [
        "TGZ"
      ]
    }
  ],
  "workflowPresets": [
    {
      "name": "ci-clang",
      "steps": [
        {
          "type": "configure",
          "name": "ci-clang"
        },
        {
          "type": "build",
          "name": "ci-clang"
        },
        {
          "type": "test",
          "name": "ci-clang"
        },
        {
          "type": "package",
          "name": "ci-clang"
        }
      ]
    },
    {
      "name": "ci-gcc",
      "steps": [
        {
          "type": "configure",
          "name": "ci-gcc"
        },
        {
          "type": "build",
          "name": "ci-gcc"
        },
        {
          "type": "test",
          "name": "ci-gcc"
        },
        {
          "type": "package",
          "name": "ci-gcc"
        }
      ]
    }
  ]
}