# Testing with MPI+OpenMP+\`ctest -j\`

**URL:** https://discourse.cmake.org/t/testing-with-mpi-openmp-ctest-j/8982
**Category:** Usage
**Tags:** tool:ctest
**Created:** [September 15, 2023, 9:39am UTC](https://discourse.cmake.org/t/testing-with-mpi-openmp-ctest-j/8982 "2023-09-15T09:39:52Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Lecris](https://discourse.cmake.org/user_avatar/discourse.cmake.org/lecris/32/3193_2.png) [@Lecris](https://discourse.cmake.org/u/Lecris)
#### Post date: [September 15, 2023, 9:39am UTC](https://discourse.cmake.org/t/testing-with-mpi-openmp-ctest-j/8982/1 "2023-09-15T09:39:52Z")

</div>

I have recently encountered a project that hard-codes the values passed through `MPIEXEC_NUMPROC_FLAG` and the `OMP_NUM_THREADS` so that they add-up to `$(nproc)`. But on the packaging side, `-j 2` is automatically added, thus oversubscribing the CPU cores. Is there an efficient way to define this execution dynamically? Maybe with generator expressions, or some internal ctest logic?

```cmake
if (USE_MPI)
  if (TEST_MPI_RANKS STREQUAL "auto")
    include(ProcessorCount)
    ProcessorCount(nproc)
    math(EXPR num_ranks "(${nproc}+${TEST_OMP_THREADS}-1)/${TEST_OMP_THREADS}"
    )# get 1/$TEST_OMP_THREADS the number of procs (rounded up)
  else ()
    set(num_ranks ${TEST_MPI_RANKS})
  endif ()
  message(
    "Tests will run with ${num_ranks} MPI ranks and ${TEST_OMP_THREADS} OpenMP threads each"
  )
endif ()

```

---

<div class="post-metadata">

### Author: ![scivision](https://discourse.cmake.org/letter_avatar_proxy/v4/letter/s/a87d85/32.png) [@scivision](https://discourse.cmake.org/u/scivision)
#### Post date: [September 15, 2023, 1:25pm UTC](https://discourse.cmake.org/t/testing-with-mpi-openmp-ctest-j/8982/2 "2023-09-15T13:25:00Z")

</div>

I do have several projects that run MPI tests where I have to be careful about the number of MPI workers because some of the tests require a minimum number of workers or they’ll fail.  
I use resource\_lock test property on each of the MPI tests. That way I know each MPI test will run by itself. You could also use the run\_serial property.

---

<div class="post-metadata">

### Author: ![Lecris](https://discourse.cmake.org/user_avatar/discourse.cmake.org/lecris/32/3193_2.png) [@Lecris](https://discourse.cmake.org/u/Lecris)
#### Post date: [September 15, 2023, 1:36pm UTC](https://discourse.cmake.org/t/testing-with-mpi-openmp-ctest-j/8982/3 "2023-09-15T13:36:50Z")

</div>

One confusing thing is that in a lot of the places it is mentioned `ctest_test` which is part of the `cdash`. It is unclear what properties can be used directly and not. @scivision have you used cdash, and is it worth it? Is it worth implementing even if the results are not uploaded.

What I would like to ultimately do is run multiple (parallel) tests in parallel since I have noticed that the tests do not scale very well, and there are various short execution tests that can be finished in parallel. For that a mechanism to dynamically define the number of MPI processes is necessary. I know that part of the mechanism is possible since it is possible to dynamically define tests (e.g. catch2), but I haven’t investigated how to adjust that interface for MPI+OpenMP tests.