parallel build (2) : add_subdirectory to a non-subdirectory

Code
#1 
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../myStaticLib ${BINARY_OUTPUT_PATH})
add_executable(${EXE_NAME} ${PROJECT_SOURCE_DIR}/exe.cpp)

Always fails if a parallel build is triggered (e.g. make -j N), and the the build in the subdirectory takes longer than the build of the exe.

How to wait for the results of add_subdirectory before, the build of add_executable is started?

#2

Such setup works quite normally for me. Please show how above_lib is defined in the file ${SRC}/../above_lib/CMakeLists.txt.

#3

Changed the above code a little. The above_lib which is myStaticLib is generated over a huge cmake file, which I would not paste here, For me to things happening:
Due to the parallel build (i.e. the on several cores), the exe files are build much quicker, and thus it starts linking against the myStaticLib before it is compiled. Thus it complains about non-existing lib, which is supposed to be linked to.
What information else, I could provide else ?

#4

That is not how CMake-generated buildsystems behave, if you have set up your dependencies correctly.

Please show:

  • The add_library() call which creates the static library
  • The target_link_libraries() call which makes the executable link against the static library.

Ideally, you’d trim your buildsystem down to an SSCCE and post that, but the above calls could be enough, so let’s start with them.

#5

In the meanwhile, I could figure out what is the problem.

To mention also: its a Linux CMake build.

I had instead of this

  • A set(MY_STATIC_LIB_NAME myStaticLib)

used this

  • B set(MY_STATIC_LIB_NAME libmyStaticLib.a)

The call below (target_link_libraries) functions with both variants.

target_link_libraries(${EXE_NAME} PUBLIC ${MY_STATIC_LIB_NAME})

But the build behaves differently:

  • A. Everything works fine, i.e. the lib is build in parallel, and the linker is called after its been built.
  • B. CMake didn’t make a connection between libmyStaticLib.a and myStaticLib from the add_subdirectory call. Thus the two were build separately. The linker stripped though the libmyStaticLib.a to myStaticLib and issued an error (/usr/bin/ld: cannot find -lmyStaticLib) of not finding the library.

Two final questions

  1. Is the above CMake behavior a bug or a feature ?
  2. How does CMake internally handles build dependencies, so that it wont call the linker before everything is build, i.e. how is build order handled?
#6

Feature.

target_link_libraries(myExe myLib) tells CMake to link the executable myExe to the library myLib, which is a CMake target built by this project. This tells CMake to ensure target dependencies and build order are correctly maintained etc. You don’t want to have to change the target_link_libraries call just because you later decide to turn myLib from a static library into a shared one (see also BUILD_SHARED_LIBS), or to accommodate the differences between Windows, Linux, MacOSX, etc.

In contrast, target_link_libraries(myExe libmyLib.a) tells CMake to link the executable myExe against a pre-existing library libmyLib.a, which is totally independent of the current buildsystem. You’d use this to link to a system library or a 3rd-party one (and you’d want to use an absolute path for the latter to ensure correct rebuilds when the library changes).

By correctly encoding the dependencies in the generated buildsystem. When generating Makefiles, it will encode the dependency in the build rule, listing the library in the exeutable’s rule’s dependencies. When generating a Visual Studio solution, it will populate the ProjectSection(ProjectDependencies) section of the .sln file accordingly. And so on for other generators.

1 Like
#7

By correctly encoding the dependencies in the generated buildsystem. When generating Makefiles, it will encode the dependency in the build rule, listing the library in the exeutable’s rule’s dependencies.

Does it mean CMake take care of build order, or the build system is responsible for it?

#8

CMake generates the buildsystem files and its work is done. It does not run as part of the build(1). I would answer your question thus: “CMake generates the buildsystem files in such a way that when the build tools run, they will follow the correct build order.” I don’t know which one (or both) you’d like to consider “responsible” given this.

(1) Even if you invoke your build as cmake --build, it’s not CMake doing any actual building. It just executes the correct build driver (devenv.exe, msbuild.exe, make, ninja, …) based on the generator which had been used to create the buildsystem.

1 Like
#9

The build order is driven by the build tool as explained by @Angew not CMake.
The effective order may (and will) vary in case of parallel build, e.g. ninja is processing a DAG of dependencies so that any node which is free will be built as soon as computing resource is available.

CMake is translating the dependencies explicitly (like in custom_command DEPENDS) or implicitly (like with target_link_libraries) specified by the developer in the CMakeLists.txt.

From my experience when your parallel build fails this means that you (as a developer) forgot to specify dependency at CMake level so that CMake cannot put it right in the generated buildsystem.

I that case you have to investigate the dependencies.
I know two means:

with those means you will be able to dump the dependencies seen/processed by CMake.

1 Like