In the CI build environment I am using (Jenkins) the project I am building sometimes gets odd test failures.
The test failures don’t appear to be caused by the unit test itself but instead seem to be an error in CTest.
Here is an example test failure log, as recorded with the CTest ‘-VV’ comman line option:
86: Running main() from /Users/mybuild/jenkins/workspace/my-merge-request-123/src/mySourceFile.test.cpp
86: Note: Google Test filter = MyControllerTest.DeviceStateUpdateTest
86: [==========] Running 1 test from 1 test suite.
86: [----------] Global test environment set-up.
86: [----------] 1 test from MyControllerTest
86: [ RUN ] MyControllerTest.DeviceStateUpdateTest
86: [ OK ] MyControllerTest.DeviceStateUpdateTest (0 ms)
86: [----------] 1 test from MyControllerTest (0 ms total)
86:
86: [----------] Global test environment tear-down
86: [==========] 1 test from 1 test suite ran. (0 ms total)
86: [ PASSED ] 1 test.
86/105 Test #86: MyControllerTest.DeviceStateUpdateTest ...................................***Exception: Illegal 0.02 sec
Any help would be appreciated.
Is there an CTest option to not report errors when the test fixture is being closed?
I’m using CMake 3.25.0. I’ve searched through the CMake issue tracker and this discourse to find similar issues, but found nothing.
Do you build on one machine and test on another? Compilers can default to -march=native and if you build/test on a rotation of machines, you might build on a new processor and test on an old one. We ended up passing -m flags to turn off features in our CI builds to get around this (which is wanted anyways as they build our binaries and we don’t want to require new processors unnecessarily just because of what machine happened to build it).
No, the builds and tests are run on the same build agent computers, back to back. The tests execute after the build is complete.
Interestingly, I was able to get the tests to pass consistently on Windows using “add_tests” instead of “gtest_discover_tests”. This change causes the gtest executable program to be launched once per executable, instead of once per unit test.
On macOS, this change didn’t improve the failure rate. I got a slightly different error. I took it one step further, and ran the unit tests without CTest at all, and ran the gtest executables directly. I’m still getting an error. Now consistently though. It makes me wonder if there is some Rosetta issue or universal binary issue occurring here.
The original CTest error was: ***Exception: Illegal 0.02 sec
The new CTest error is:
Subprocess aborted***Exception: 0.61 sec
The consistent GTest execuable error is: ../build_macos_x86.sh: line 97: 87955 Abort trap: 6
Now that the error is consistent, it may be easier to identify a root cause.