Avoid duplicate linking to avoid Xcode 15 warnings

Code
1

Xcode 15 includes a new linker which complains when passing a static library more than once to the linker.

What do we need to pay attention to to prevent CMake from duplicating libraries? What is a convenient way to determine why a library got duplicated?

The motivation for this question is this issue with igraph: "ignoring duplicate libraries" and "duplicate LC_RPATH are deprecated" linker warnings · Issue #2394 · igraph/igraph · GitHub (now resolved).

Here’s a minimal example that demonstrates how a static library may get included twice on the linker line. It took much trial and error to distil down a much more complex system to this example.

cmake_minimum_required(VERSION 3.18)
project(Foo)

add_library(one one.c)

add_library(two OBJECT two.c)
target_link_libraries(two PRIVATE one)

add_executable(main main.c)
target_link_libraries(main PRIVATE one)
target_link_libraries(main PRIVATE two)

This links libone.a twice. If we reverse the order of the last two lines, as in the following code block, then there is no duplication.

target_link_libraries(main PRIVATE two)
target_link_libraries(main PRIVATE one)

Is this by design? Is there a simple way to avoid duplication without reversing the order? More importantly, is there a simple way to determine where the duplication came from (in a large CMake-based project) and what specifically needs to be reordered?

2

On many platforms, linkers do not re-scan static libraries from earlier on the link line to satisfy symbol dependencies of those later on the link line. target_link_libraries(two PRIVATE one) says that two’s symbols may depend on symbols from one. In order to satisfy this, CMake ensures that one appears on the link line after two. However, you also told CMake that main needs to link to one and then two. We always preserve the direct link dependencies in order, and then add transitive dependencies afterward. The only way to satisfy both requirements is to generate one two one on the link line.

IIRC, Apple’s linker does re-scan static libraries, so repeating them is not needed to satisfy dependencies. This is similar to how linkers treat shared libraries, which we do de-duplicate. That code could be updated to de-duplicate static libraries too when using the Apple linker.

1 Like
3

Thanks, this makes it clear what is going on.

4

This would be useful. I am encountering a situation where the duplicate library is -lc++, which is implicit, and I can’t seem to figure out any reordering of CMake commands that helps.

5

Meanwhile try adding

target_link_options(main PRIVATE LINKER:-no_warn_duplicate_libraries)

to suppress the warning, though it may take some investigation to determine the proper condition for whether the new Apple linker is used.

2 Likes
6

This is now tracked as CMake Issue 25297.

1 Like
7

I read through this issue and I wasn’t 100% sure what the ultimate fix is. If I am building software targeting macOS and Linux, is the goal to allow the direct and transitive libs to have duplicate entries on Linux and macOS and just suppress the warnings on macOS or is it to de-duplicate the full combined set of direct and transitive libs so both systems don’t have any duplication? I apologize if this is obvious.

8

It is to de-duplicate. CMake MR 8946 added de-duplication on relevant platforms, starting in CMake 3.29.

9

Ultimately, I have the “same” scenario that is described above with some small differences (information listed below). Things are not de-duplicated for me. I am on macOS 14.6.1 utilizing CMake v3.29.5, Clang 17.0.6, and OpenMPI 4.1.6 from MacPorts.

  1. libB and libC depend on libA.
  2. appZ depends on libA and libB.
    1. libA is on the link line twice.
  3. appY depends on libA, libB, and libC.
    1. libA is on the link line twice (so the transitive libraries appear to be de-duplicated by CMake).

I haven’t found any link specification order that results in de-duplication. This differs from the example in the aforementioned issue.

  • I tried libA before libB and libC.
  • I tried libA after libB and libC.

What do I need to do? The MR points to the following, although the poster seems to indicate it doesn’t work. Is this syntax correct?

cmake_policy(SET CMP0156 NEW)
set(CMAKE_C_LINK_LIBRARIES_PROCESSING ORDER=REVERSE UNICITY=ALL)
set(CMAKE_CXX_LINK_LIBRARIES_PROCESSING ORDER=REVERSE UNICITY=ALL)
10

With this project:

cmake_minimum_required(VERSION 3.29) # sets CMP0156 to NEW
project(Discourse9084 C)

add_library(A a.c)
add_library(B b.c)
target_link_libraries(B PUBLIC A)
add_library(C c.c)
target_link_libraries(C PUBLIC A)
add_executable(Y y.c)
target_link_libraries(Y PRIVATE A B C)
add_executable(Z z.c)
target_link_libraries(Z PRIVATE A B)

On Linux I get link lines:

cc ... -o Y  libA.a  libB.a  libC.a  libA.a
cc ... -o Z  libA.a  libB.a  libA.a

and on Apple I get link lines:

cc ... -o Y  libB.a  libC.a  libA.a
cc ... -o Z  libB.a  libA.a

The latter are de-duplicated as expected because CMake (after CMP0156) knows that Apple’s linker re-scans automatically.

The CMAKE_<LANG>_LINK_LIBRARIES_PROCESSING variable is an undocumented implementation detail that CMake sets internally/automatically based on what linker is being used.

11

Just to provide my two cents here, I am surprised by this behavior.

I think the core problem here is that CMake does not support the ability to order the libraries according to their dependency order as:

cc ... -o Y  libC.a  libB.a  libA.a

instead of the manual project ordering:

cc ... -o Y  libA.a  libB.a  libC.a  libA.a

for the example above.

This is a simple graph problem to figure out the ordering of the libraries according to the dependencies and only list the libraries once. (I implemented that in old TriBITS in CMake code. That should be easy to implement in C++ with proper graph data-structures and algorithm.)

Any reason why CMake can’t implement library ordering based on dependencies to avoid this problem all together (unless there really are circular dependencies in the libraries and they can do it manually in your CMakeLists.txt files)? If that is a new global or target property that can be set, then fine. My guess is that 95% of customers would prefer CMake to reorder the libraries and remove duplicates. That will yield shorter link lines and faster links.

NOTE: I think the reason I never noticed this behavior this because TriBITS does auto linking and takes into account package dependency order when it does that. I can guarantee if you switch to manual linking, this problem will come up.

12

CMake computes a link line by starting with the exact list provided by the project and then adding transitive dependencies it doesn’t reflect. target_link_libraries(Y PRIVATE A B C) says to link Y with A B C first: CMake adds another A to satisfy B -> A. This approach gives projects control over the order for cases where it matters.

I’ve opened CMake Issue 26271 to propose an option to re-order direct dependencies to honor their transitive dependencies.

1 Like