Control build parallelism within ExternalProject_Add

Code
#1

Hi,

I am working on a project to build&package 3rd party dependencies for an application, by calling ExternalProject_Add exclusively. This runs in linux vm-s using both Ninja and Unix Makefile generators.

The problem:

  • Ninja automagically detects parallelism. Well, in practice it detects the number of host CPUs instead of the ones allocated to the VM → overcommitting and competing for resources → very slow build
  • Unix Makefile runs in serial by default

What I tried:

  • added parallel level, no effect:
      ExternalProject_Add(
      CMAKE_ARGS -DCMAKE_BUILD_PARALLEL_LEVEL=${NCORES} -DENV{CMAKE_BUILD_PARALLEL_LEVEL}=${NCORES}
      ...
      )
  • also tried to set the job pool for Ninja, by post-adjusting the target, also no effect:
      set_property(GLOBAL PROPERTY JOB_POOLS builder_pool=${NCORES})
      ExternalProject_Add(${target_name} ...)
      set_property(TARGET ${target_name} PROPERTY JOB_POOL_COMPILE builder_pool)
      set_property(TARGET ${target_name} PROPERTY JOB_POOL_LINK builder_pool)

So the question is, how can I override the number of cores for the individual builds? I only need to support Unix MakeFile and Ninja generators, so an if (CMAKE_GENERATOR EQUALS “Ninja”) … else … kind of solution is still acceptable for me (but something generic would be OK too :wink: ).

#2

It’s not really supported well unfortunately. I’ve done some work to make this happen in this project, but it’s never been upstreamed.

See [this code[(https://gitlab.kitware.com/paraview/common-superbuild/-/blob/master/cmake/SuperbuildExternalProject.cmake#L16-L32) (and the variable uses later in the file) for how it’s done. You can probably get away by manually specifying BUILD_COMMAND or something.

#3

At the end figured out something simpler. One can chain the cmake commands to set CMAKE_BUILD_PARALLEL_LEVEL and this works (maybe I am doing something wrong but passing --parallel 2 is failing with argument “2” not understood).

ExternalProject_Add(
...
BUILD_COMMAND ${CMAKE_COMMAND} -E env CMAKE_BUILD_PARALLEL_LEVEL=${NCORES} ${CMAKE_COMMAND} --build .
)

This is not a general solution but works for me because it’s always --build . for my project. TBH, things would be easier if the commands could be modified for example with ExternalProject_Get_Property/ExternalProject_Set_Property. Something like this:

# Do not specify BUILD_COMMAND, so the default will be generated
ExternalProject_Add(${target_name} ...)
ExternalProject_Get_Property(${target_name} BUILD_COMMAND)
ExternalProject_Set_Property(${target_name} BUILD_COMMAND ${CMAKE_COMMAND} -E env CMAKE_BUILD_PARALLEL_LEVEL=${NCORES} ${BUILD_COMMAND})
#4

See also Efficiency issues with ExternalProject sub-builds under Ninja