Expected failures on master?

Development
1

At first I thought there was something wrong in my branch, but just running the tests on master I’m seeing failures, e.g. RunCMake.NinjaMultiConfig fails for me as follows. Am I missing some prerequisite for successful testing?

CMake Error at /Users/dave/src/cmake/Tests/RunCMake/NinjaMultiConfig/RunCMakeTest.cmake:519 (message):
  RunCMake_TEST_FAILED:Actual files did not match expected

  Expected:

    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/CMakeFiles/exe.dir/Debug/exe_autogen/mocs_compilation_Debug.cpp.o
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/CMakeFiles/exe.dir/Debug/qt5.cxx.o
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/Debug/exe
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/deps_Debug
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/include_Debug/moc_qt5.cpp
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/include_Debug/moc_qt5.cpp.d
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/mocs_compilation_Debug.cpp
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/timestamp_Debug

  Actual:

    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/.qt/QtDeployTargets-Debug.cmake
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/.qt/QtDeployTargets-RelWithDebInfo.cmake
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/.qt/QtDeployTargets-Release.cmake
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/CMakeFiles/exe.dir/Debug/exe_autogen/mocs_compilation_Debug.cpp.o
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/CMakeFiles/exe.dir/Debug/qt5.cxx.o
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/Debug/exe
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/deps_Debug
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/include_Debug/moc_qt5.cpp
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/include_Debug/moc_qt5.cpp.d
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/mocs_compilation_Debug.cpp
    /Users/dave/src/cmake/build-ninja-multi/Tests/RunCMake/NinjaMultiConfig/QtX-Debug-in-Debug-better-graph-ON-build/exe_autogen/timestamp_Debug
2

There is a nightly CI schedule with a job covering Ninja Multi-Config in master and release branches every night. It’s been clean for a long time.

Local configuration and version differences may play a role. I’m not familiar with what could cause .qt/QtDeployTargets-*.cmake files to appear in the build tree.

3

Okay; are there instructions for setting up a reliable local build/test environment somewhere?

This isn’t the only one failing, for example:

116/729 Test: CustomCommand
Command: "/Users/dave/src/cmake/build-ninja-multi/bin/Release/ctest" "--build-and-test" "/Users/dave/src/cmake/Tests/CustomCommand" "/Users/dave/src/cmake/build-ninja-multi/Tests/CustomCommand" "--build-two-config" "--build-generator" "Ninja Multi-Config" "--build-makeprogram" "/opt/homebrew/bin/ninja" "--build-project" "CustomCommand" "--build-exe-dir" "/Users/dave/src/cmake/build-ninja-multi/Tests/CustomCommand/bin" "--build-options" "--test-command" "CustomCommand"
Directory: /Users/dave/src/cmake/build-ninja-multi/Tests
"CustomCommand" start time: May 21 10:35 PDT
Output:
----------------------------------------------------------
Internal cmake changing into directory: /Users/dave/src/cmake/build-ninja-multi/Tests/CustomCommand
Error: cmake execution failed
Configuring done (0.0s)
CMake Error at CMakeLists.txt:595 (add_custom_command):
  OUTPUT containing a "#" is not allowed.


Generating done (0.0s)
CMake Generate step failed.  Build files cannot be regenerated correctly.

<end of output>
Test time =   0.08 sec
----------------------------------------------------------
Test Failed.
"CustomCommand" end time: May 21 10:35 PDT
"CustomCommand" time elapsed: 00:00:00
----------------------------------------------------------
4

That failure looks like the build isn’t up to date w.r.t. the test sources.

6

So it turns out if I brew uninstall --ignore-dependencies qt the NinjaMultiConfig test passes (I had qt6 installed). I had to --ignore-dependencies because some of my installed software depends on it. Is that a bug in the test that I should report?