[CTest] best practice for running tests on a cluster

Usage
1

I’m wondering what the best way to run tests on a cluster would be when I use CTest. The objective is to run a number of tests on the compute nodes of a cluster that has the SLURM resource aanager. Any tips are welcome :slight_smile:

1 Like
2

@chuckatkins? My gut says that the test commands would need to use slrum, but maybe there’s some other trick available.

3

Ok, I suspected as much as googling did not turn up much.

Some testing shows that it is easy to do with an interactive job, e.g. to run an 8-core “test_command” on a single node:

add_test(NAME test_srun 
         COMMAND srun --nodes 1 --output test_output-%A.out 
                 mpirun -n 8 <test_command> <test_options>)
4

Hi @menno,

If you run individual tests on a cluster using slurm, be careful about waiting times, timeouts and other problems.
Also some clusters have different environments on compute nodes, so you may need to load modules etc. This can be tedious to get right and maintain in single commands.

At preCICE, we found that it’s way easier to run the entirety of ctest in a single job script. This also simplifies getting the setup working before moving to more involved setups.

Best
Frédéric

1 Like