We are running into an issue with ctest, where tests can time out after running for a time way shorter than their
TIMEOUT property, sometimes as soon as a few seconds after starting.
We are hitting this only in specific conditions, pretty much only when running the full test suites (around 3000 tests, rather CPU-intensive, that take approximately 30-minute long with
CTEST_PARALLEL_LEVEL=36) of a codebase of ours, in a CI pipeline, which runs on Linux, in a docker. Out of the 3000, a few tests (2 or 3) usually time out, and a lot sooner than expected.
To give a few examples of what I’m talking about:
- I have a test with a
TIMEOUTproperty of 1200, and yet it timed out multiple times, after 75.09, 147.96, 49.78, 108.15, 106.72, 59.85, and 95.28 seconds.
- I also have quite a few instances of tests with a
TIMEOUTvalue of 240, that timed out after running anywhere between 600ms and 1m 1s 710ms.
I am struggling to reproduce it in simpler conditions, and I am running out of ideas. Does this sounds anything like a known issue? (I could not find anything relevant on gitlab or discourse.) Or does anyone have an idea of what conditions could make this happen ?
We are using cmake 3.18.4, and I also reproduced the issue with 3.22.1.