file(GET_RUNTIME_DEPENDENCIES) issues

Usage
1

EDITED:

This thread has become more about issues/bugs with file(GET_RUNTIME_DEPENDENCIES) which is completely different than the intended topic. To not confuse future readers I’ve edited the issue.

2

@kyle.edwards Do we have examples of file(GRD) usage? There is the test suite, but that is probably more on the contrived side of things I imagine. Is your common-superbuild migration branch available anywhere (even as a rough draft)?

3

GET_RUNTIME_DEPENDENCIES has quirks and issues in cmake 3.17.2 so an answer to your question will depend on the build platform.
Short story:
On Linux it does not work as expected so I needed to implement something naiv. The following code is a excerpt from my code and maybe lacking variable declarations, functions definitions, … and elegance but the gist is hopefully clear.

# GET_RUNTIME_DEPENDENCIES does not work on Linux using cmake 3.17.2
#   it does not honour the paths given by ldd and searches in system paths itself.
#   in doing so it picks up 32 Bit system libs for 64 Bit builds
# this is a naive implementation that uses ldd 
if (DEFINED ENV{LD_LIBRARY_PATH})
  message (WARNING "LD_LIBRARY_PATH is defined. This may lead to the installation of library versions that were not used at link time.")
endif()
foreach (lib ${all_libs})
  execute_process(COMMAND ldd ${lib} OUTPUT_VARIABLE ldd_out)
  string (REPLACE "\n" ";" ldd_out_lines ${ldd_out})
  foreach (line ${ldd_out_lines})
    string (REGEX REPLACE "^.* => | \(.*\)" "" pruned ${line})
    string (STRIP ${pruned} dep_filename)
    if (IS_ABSOLUTE ${dep_filename})
      is_system_lib (${dep_filename} sys_lib)
      if (sys_lib EQUAL 0 OR INSTALL_SYSLIBS STREQUAL "true")
        list (FIND dependencies ${dep_filename} found)
        if (found LESS 0)
          list (APPEND dependencies ${dep_filename})
        endif()
      endif()
    endif()
  endforeach()
endforeach()

On windows it works but has problems with API_SETS. Small code snippet

 if (CMAKE_HOST_SYSTEM_NAME STREQUAL "Windows")
#[[
The following statements are a workaround of problems that cmake 3.17 has with Windows API sets.
Reference: https://ofekshilon.com/2016/03/27/on-api-ms-win-xxxxx-dll-and-other-dependency-walker-glitches/
#]]
LIST(APPEND pre_exclude_regexes "api-ms-.*") # windows API
LIST(APPEND pre_exclude_regexes "ext-ms-.*") # windows API
LIST(APPEND pre_exclude_regexes "ieshims.dll") # windows API
LIST(APPEND pre_exclude_regexes "emclient.dll") # windows API
LIST(APPEND pre_exclude_regexes "devicelockhelpers.dll") # windows API

LIST(APPEND post_exclude_regexes ".*WINDOWS[\\/]system32.*") # windows system dlls

file(GET_RUNTIME_DEPENDENCIES
    RESOLVED_DEPENDENCIES_VAR dependencies
    LIBRARIES ${all_libs}
    DIRECTORIES ${TBT_PATH_EXTERNAL}
    CONFLICTING_DEPENDENCIES_PREFIX conflicts
    PRE_EXCLUDE_REGEXES ${pre_exclude_regexes}
    POST_EXCLUDE_REGEXES ${post_exclude_regexes}
    POST_INCLUDE_REGEXES ${post_include_regexes}
    )
list (LENGTH conflicts_FILENAMES num_files_conflict)
if (${num_files_conflict} GREATER 0)
  message(WARNING "dependencies of ${lib} found at multiple locations will not be installed.")
  foreach (lib_conflict ${conflicts_FILENAMES})
    message(STATUS "${lib_conflict} found at: ${conflicts_${lib_conflict}}")
  endforeach()
endif()

endif()

4

Hmm. This is news to us. AFAIK, it is working as expected, though multilib or multiarch installations could certainly confuse it. Note that cross-compilation is not supported at the moment (see this issue). Could you please file an issue about what is going on? @kyle.edwards

See this question for how I think CMake should provide these pre-exclude regexes.

5

I suspect that’s mostly an issue of rpaths (or lack thereof), and the fact that shared libraries can’t just be copied around willy-nilly on Linux, no?

For example, /usr/bin/bash as installed on my Fedora 33 system doesn’t really have a runtime dependency on the file /lib64/libc.so.6 at all — it has a dependency on the name libc.so.6. Only the system’s dynamic loader is equipped to resolve that name usefully. So just knowing the path to /lib64/libc.so.6 doesn’t really do you a whole lot of good all by itself.

6

I am not a Linux specialist, so please take what I am writing as my impression and not my definite view.

  1. We are packaging some libraries from the build system in order to make the software usable on “many systems” and without needing requiring the user to install loads of stuff he might not even be able to get via the package manager of his distribution. As such it may be viewed as something similar as AppImage or Flatpack.
  2. The libraries we have been packaging prior to using cmake are from the FHS secondary hierarchy libs ( Filesystem Hierarchy Standard - Wikipedia).
  3. To my knowledge the so depencies reference a full path the shared libraries it has been linked against.The runtime loader uses precedence whilst loading: LD_LIBRARY_PATH, RPATH, FHS standard location.
  4. During the build LD_LIBRARY_PATH is not used.
  5. Whilst packaging the exact libraries that have been used whilst linking need to packaged. This means that LD_LIBRARY_PATH may not be used and that it is not neccassary to reimplement what the buildtime linker does whilst selecting the “FHS standard location”. This selection is already visible in the output of ldd.
  6. What I think is wrong in GET_RUNTIME_DEPENDENCIES is
    a. it evaluates LD_LIBRARY_PATH
    b. it (uncessesarily) reimplements the buildtime linker but in doing so picks up incorrect versions of the shared objects, e.g. 32 Bit for a 64 Bit build.
7

Well, that’s certainly possible. I’m more curious whether it is our code which breaks the “works as expected” or there’s a misunderstanding of how ELF library searching and loading works.

I believe you mean runtime linker, but the thing is that ldd gives the full recursive list of libraries found. That interferes with the cross-platform logic we want to use for the pre- and post- include and exclude regexes. We can certainly inspect the binaries we find and do architecture matching to exclude libraries which are not viable (though there are various reasons a library might be unsuitable, architecture matching is probably fine for 90% of the use cases). (Like I said, cross-compilation is definitely not yet supported and multi-lib and multi-arch setups could use some more testing.)

What I think you want to do is to use $ORIGIN as the (basis of) your install time rpath. You’ll want to edit libraries you copy into your package as well (this kind of stuff is planned for a future API by CMake, but it is much more complicated because the platform differences are far greater in this region). file(GRD) is based on the strategy we’ve been using for creating ParaView packages for the past few years which has been working really well.

8

It’s a bit more complex than that, even — the runtime loader keeps a cache (maintained by ldconfig) which maps between library symbols and the .so files that provide them. The cache is normally populated by scanning all of the shared library files in any of the directories configured in /etc/ld.so.conf (or, if utilized by the Linux distro, by the drop-in files added to /etc/ld.so.conf.d/), in addition to the predefined standard paths.

But you’re right that in the build tree, libraries not linked through standard -llibname means (such as most IMPORTED targets from other CMake projects’ Config.cmake files) will be referenced by their full disk path. Dynamic linking cares not for such things, though (only the library SONAME will be stored in the binary’s symbol table, even if the compiler is given a full path), so if the file isn’t in a standard system dir, its location will also be set as an RPATH on the build-tree binary so that the library is preferentially loaded from there.

In its default install mode, CMake removes all RPATHs, which means that a binary’s shared library dependencies can and will change from the build-time linking if there are different versions of the same shared library installed on the system, compared to what it was linked with when it was built.

There are also several target properties that affect all of that, some in non-obvious / confusing ways.

  • BUILD_WITH_INSTALL_RPATH will avoid the build-tree-only RPATHs that make compiled libraries and executables runnable from the build tree even if they reference uninstalled libs. Which mostly just serves to make your binary unable to be run from the build tree unless it’s only linked with installed libraries. (OTOH, if using an INSTALL_RPATH that can be a good way of verifying that it’s set correctly.)
  • In recent CMake versions, BUILD_RPATH_USE_ORIGIN will cause any linking within the build tree to use relative paths. Since it doesn’t affect any linking outside the build tree. It’s really only useful for making your build tree relocatable, on the odd chance you need to do that.
  • INSTALL_RPATH_USE_LINK_PATH adds non-standard dependency locations to the installed RPATH of any binary it’s set on, instead of the default blanking. Which can be extremely helpful, though it has one major shortcoming: It only works on outside dependencies. If your project builds, say, a shared library and an executable linked with it, and you install that project in a non-standard location, your executable’s still not going to be able to find your own shared library unless you set an appropriate INSTALL_RPATH. (Or if something like an INSTALL_RPATH_USE_ORIGIN is added to CMake.)

Yeah, it will affect the results if set, same as any tool that uses the system loader — including ldd. On Linux GET_RUNTIME_DEPENDENCIES is just a wrapper for calls to objdump -p. (By default, anyway; that can be changed by setting CMAKE_GET_RUNTIME_DEPENDENCIES_COMMAND.) In this context objdump only really differs from ldd in that it’s not recursive — CMake does its own recursion, though. The output of objdump -p is then parsed for RPATH, RUNPATH, and NEEDED lines, to collect dependency details.

Ouch. Yeah, when I said the “only difference” between objdump and ldd was recursion, I was wrong. objdump also has no native awareness of architecture, the tool has to account for that itself. GET_RUNTIME_DEPENDENCIES fails to, it seems, so I’m seeing the same thing here.

In fact, on my system (CMake 3.18.4 on Fedora 33 x86_64), it seems GET_RUNTIME_DEPENEDENCIES will prefer the 32-bit version of any multiarch library installed on the system, maybe just because the path is shorter.

So, e.g. an installed /lib/libQt5core.so.5 gets returned for a 64-bit Qt executable that’s really linked with /lib64/libQt5core.so.5. And that’s true even if it also returns /lib64/libQt5Gui.so.5 because /lib/libQt5Gui.so.5 is not installed. So it not only picks the wrong arch, but has no protections against mixing 32-bit and 64-bit libraries. That’s… kind of a mess.

I’ll also add that GET_RUNTIME_DEPENDENCIES is DOG slow, holy crap. I think it’s actually executing ldconfig and re-parsing its entire output for each individual dependency encountered. Which, on a desktop system where ldconfig -N -p -X |wc -l shows 5428 lines, is unbearably glacial.

For my executable with these dependencies:

Dynamic Section:
  NEEDED               libpthread.so.0
  NEEDED               libMagick++-6.Q16.so.8
  NEEDED               libMagickWand-6.Q16.so.6
  NEEDED               libMagickCore-6.Q16.so.6
  NEEDED               libjsoncpp.so.24
  NEEDED               libQt5Widgets.so.5
  NEEDED               libQt5Gui.so.5
  NEEDED               libQt5Core.so.5
  NEEDED               libavcodec.so.58
  NEEDED               libavdevice.so.58
  NEEDED               libavformat.so.58
  NEEDED               libavfilter.so.7
  NEEDED               libavutil.so.56
  NEEDED               libpostproc.so.55
  NEEDED               libswscale.so.5
  NEEDED               libswresample.so.3
  NEEDED               libavresample.so.4
  NEEDED               libgomp.so.1
  NEEDED               libzmq.so.5
  NEEDED               libstdc++.so.6
  NEEDED               libm.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6

a ONE MINUTE FORTY-FIVE SECOND pause occurs during cmake --install, due to this code which does nothing but display the results:

install(CODE [[
  file(GET_RUNTIME_DEPENDENCIES
    EXECUTABLES $<TARGET_FILE:myexe>
    RESOLVED_DEPENDENCIES_VAR _r_deps
    UNRESOLVED_DEPENDENCIES_VAR _u_deps
  )
  foreach(_file ${_r_deps})
    message(STATUS "Dependency: ${_file}")
  endforeach()
]])

…and after all that wait, then shows a whole bunch of incorrect 32-bit library paths mixed in with the 64-bit ones.

9

FTR: The arch issue comes about because the output of ldconfig will list all libs installed in any architecture, and at least on Fedora systems they all have the same names, just at different paths. So, e.g. this is how they’re listed in the cache:

$ ldconfig -N -p -X |grep Qt5Core
	libQt5Core.so.5 (libc6,x86-64, OS ABI: Linux 3.17.0) => /lib64/libQt5Core.so.5
	libQt5Core.so.5 (libc6, OS ABI: Linux 3.17.0) => /lib/libQt5Core.so.5
	libQt5Core.so (libc6,x86-64, OS ABI: Linux 3.17.0) => /lib64/libQt5Core.so
10

Path length doesn’t matter; path order does.

As for speed…I haven’t noticed it with the Python implementation so much, but I also haven’t ported over yet (time…) and that does do forking for objdump for each library. There’s only one ldconfig call to get paths though (-v -N -X) and the grep you need is for '^(/.*):' (basically, the real one is a bit more specific). On my machine, that’s 3 paths, but I also don’t have any 32bit libraries installed either.

11

I really meant the “build time linker” in order to make sure I package exactly what I explicitly or implicitly linked against into the kit. This is done to make the kit less dependent on what is installed on the target system, e.g. X11.

I am pretty sure this is not the case on my system because it actually picked up libraries in /lib and /usr/lib for a 64 Bit build. ldd does not.

Yes: I change rpath to $ORIGIN for all “system libraries” during cmake install step. I use patchelf for this purpose because I did not find out how to do this using cmake.

12

I also noticed this and whilst not a reason for my naiv substitute for GET_RUNTIME_DEPENDENCIES was a pleasant side effect.

13

But all we have is the runtime linker information; the build link line is long gone by the time we get the final library or executable. If you build with mixed up RPATH information, we’re going to be just as confused as the runtime linker.

I said “can certainly inspect” in that we could, but do not currently do so.

Yep. The idea is that we’ll eventually have CMake APIs which will patch up libraries as needed using the system tools, but designing that API is difficult due to the vast differences between what platforms need.

14

Probably my implementation coincedentally works in combination of the way the .so are built and the use of ldd:

  1. Extract dependencies for each built .so using ldd (so I do not need the linker call)
  2. compute the union of all these dependencies
  3. remove all built .so from this union. This step has a filter with which I can also remove the .so in the FHS “primary hierachy” (/lib and /lib64)
  4. copy the remaining ones to the desired location in the build tree changing their RPATH to $ORIGIN
15

This is essentially what we’re doing (minus the actual install and patching), but we don’t let ldd do the recursion so that we can have consistent behavior on every platform.

16

But then cmake should never pick up .so from /lib for a 64Bit build.
Idea: Use ldd on the top level to compute all dependencies (incl. transitive) with their full path and then use objdump to figure out the direct dependencies. Then it would be possible to filter using the PRE and POST filters of GET_RUNTIME_DEPENDENCIES.

17

Well, we will once we start rejecting due to architecture mismatches in ELF binaries.

1 Like
18

Great! Then it remains to be decided if considering LD_LIBRARY_PATH and output of ldconfig is good. My gut feeling is that they are part of the “runtime” enviroment and should not be condidered in packaging. Please keep in mind that I am not a Linux wizard, so this only based on “feeling” and not on knowledge.

19

We have to use this since this is the only reliable way of determining what directories should be searched at all. Different distros use different layouts (/usr/lib (32bit on Red Hat, native on Debian pre-multiarch), /usr/lib64 (64bit on Red Hat), /usr/lib/$triple (Debian multiarch), GoboLinux layouts, etc.). I don’t think we use LD_LIBRARY_PATH (it is only mentioned in documentation snippets about ELF loading that we have).