How to tell cmake not to compile 2-3 TU-s in parallel (eg. from 100 TU-s in total), added as subprojects using add_subdirectory() - add_executable() - add_test() using -G Ninja generator.
Is there any solution or workaround for this? E.g. does any other generator support this? Thx
I describe why I need this and where the problem is.
I have ~230 TU and ~3 of them take a ridiculous amount of memory on MSYS2 and Linux g++ (like 4-5GB for these problematic TU-s) so because of these 3 TU (they are unit tests) I have to define --parallel 2 (I would normally do --parallel 10).
This problem exists for me only on g++ on MSYS2 and Linux, all other platforms like MSVC, Linux clang++, or clang-cl with MSVC are fine. So invoking CI pipelines can take 3 hours because of this (especially MSYS2 g++) because I have more combinations in the matrix. Other platforms can practically do it in a few minutes (3-20 minutes; depending on ccache).
I would like to tell CMake not to compile these 3 TUs with any other TU or compile them with max. --parallel 2. I searched the forum and found that there is no easy solution for this with Ninja.
I know that Ninja is designed to max. parallelize everything as much as possible so this is the exact opposite.
How can I overcome this or what are my options to resolve this?
If there is no native solution for this I was thinking about compiling these problematic TUs as last and making them dependent on each other so they compile synchronously. (But I don’t know how to do this or if it’s even possible).
Actually Ninja has a specific capability for this, that is also exposed in CMake: JOB_POOLS.
I have used this with CMake targets that use a large amount of RAM in the linking step. This can be dynamic by setting job pool amount of parallel workers based on system RAM.
Thx for reply, I have never heard about this, does it really work?
What if I set it like this:
# Or do I need also the default pool?
# set_property(GLOBAL PROPERTY JOB_POOLS default=10 expensive=2)
set_property(GLOBAL PROPERTY JOB_POOLS expensive=2)
# set_property(TARGET normal_unittest1 PROPERTY JOB_POOL_COMPILE default)
# set_property(TARGET normal_unittest2 PROPERTY JOB_POOL_COMPILE default)
set_property(TARGET bad_unittest1 PROPERTY JOB_POOL_COMPILE expensive)
set_property(TARGET bad_unittest2 PROPERTY JOB_POOL_COMPILE expensive)
set_property(TARGET bad_unittest3 PROPERTY JOB_POOL_COMPILE expensive)
I don’t understand how these pools will be handled, will it invoke at 12 cores at the end? 10 for default and 2 for expensive, or does it invoke these pools synchronously?
I will have to try it for sure because from docs is not clear how this works, but I’m working on something else now.
There is also a very good example in Professional CMake: A Practical Guide by Craig, but also isn’t clear how this works and how are these pools handled.
I got it, all pools are running in parallel, -jX is always saturated to max. but never exceeds defined values in pools. I was able to compile ~50% faster.
Are there any other alternatives to this?
OK you’re right, Ninja doesn’t exactly handle this, even though it was better with job pools.
You might be able to use CMake add_dependency() to artificially make the targets build mutually exclusively of each other in time. That is make the groups of targets build in a sequence you decide, one group at a time. Try that in combination with the job pools you just set up.
It would be perfect as it is now, but if there were some options how to serialize some pools, eg. invoke p2 only after the p1 finishes, it could be 1:1 as 1:n or m:n would be hell for the coder.