Issue with tests timing out way sooner than what their TIMEOUT property says

Usage
,
1

Hi there,

We are running into an issue with ctest, where tests can time out after running for a time way shorter than their TIMEOUT property, sometimes as soon as a few seconds after starting.

We are hitting this only in specific conditions, pretty much only when running the full test suites (around 3000 tests, rather CPU-intensive, that take approximately 30-minute long with CTEST_PARALLEL_LEVEL=36) of a codebase of ours, in a CI pipeline, which runs on Linux, in a docker. Out of the 3000, a few tests (2 or 3) usually time out, and a lot sooner than expected.

To give a few examples of what I’m talking about:

  • I have a test with a TIMEOUT property of 1200, and yet it timed out multiple times, after 75.09, 147.96, 49.78, 108.15, 106.72, 59.85, and 95.28 seconds.
  • I also have quite a few instances of tests with a TIMEOUT value of 240, that timed out after running anywhere between 600ms and 1m 1s 710ms.

I am struggling to reproduce it in simpler conditions, and I am running out of ideas. Does this sounds anything like a known issue? (I could not find anything relevant on gitlab or discourse.) Or does anyone have an idea of what conditions could make this happen ?

We are using cmake 3.18.4, and I also reproduced the issue with 3.22.1.

2

Are you certain the tests are being terminated with a timeout? Could it be that the OS is terminating them for some other reason, such as running out of memory, or exceeding some other kind of system limit? What does the test output show and what does ctest report as the reason for the test failures? You may need to use ctest --output-on-failure to see the test output for the failed tests.

3

Ha! Found it! We actually run the tests with the STOP_TIME option of ctest_test (to ensure the results are submitted to our CDash), and this is the timeout we are hitting.

I completely overlooked that, and I missed the line in the logs that says:

The stop time has been passed. Stopping all tests.

It would be nice if this information was shown on CDash, but otherwise everything works as intended :slight_smile:

Thank you for your reply.