Setting RPATH and not RUNPATH for executable

Code
1

I am working on a project that I would like to be able to distribute pre-compiled binaries for and I believe I’m in a scenario where I really do need to set RPATH and not RUNPATH but all the CMAKE_*_RPATH seem to be setting RUNPATH instead of RPATH. I tried using target_link_options to set it with LINKER:-rpath,"$ORIGIN/..." but am still getting RUNPATH set instead of RPATH (note the $ORIGIN needs some escaping but I didn’t bother figuring that out) so I’m not sure if this is actually on the compiler/linker side of things (I’m using GCC) rather than CMake?

Based on this stackoverflow question it sounds like RUNPATH is fully preferred over RPATH but I believe I’m in a situation where I can’t get around needing RPATH (but open to alternatives). The CMake wiki’s entry on RPATH says RUNPATH behaves exactly like RPATH except paths on RUNPATH are searched after paths on LD_LIBRARY_PATH while RPATH is searched before. It also says RPATH is ignored if RUNPATH is present so I would need to ensure RUNPATH is empty.

Scenario

I’m writing a C extension for MATLAB that wraps around a third-party C lib. On Linux, MATLAB ships with an old version of stdlibc++. The third-party C lib needs a newer version to compile correctly. When calling MATLAB, it sets up LD_LIBRARY_PATH to give the libraries it ships with preference. Because of this the dynamic linker finds the old copy of stdlibc++ and I end up with GLIBCXX_3.4.29 not found errors. I’m using CMake’s install functions to collect the RUNTIME_DEPENDENCY_SET and install it in a lib directory with the executables. Then I use (set CMAKE_INSTALL_RPATH "$ORIGIN/lib") so the linker will find those. The linker continues to find the old lib though. I can confirm removing the old lib from MATLAB’s library directory fixes the problem (though I’d rather a solution that doesn’t require the end user to touch their MATLAB install). I think my only option is to set the RPATH (I would not want to mess with LD_LIBRARY_PATH because I don’t want to impact the libraries other executables are using).

Additionally, the above wiki page indicates after setting the RPATH I can confirm it’s been set with objdump -x <executable> | grep "RPATH", this results in no output, but replacing RPATH with RUNPATH does return a list of paths.

Is there anyway to force the use RPATH? Has it been fully deprecated at this point? Is there anything else I can do to ensure the linker finds the libs shipped with my package?

2

Found it as discussed here, I was looking for the linker flag --disable-new-dtags. And the choice of setting RUNPATH vs RPATH by default does look to be a linker specific decision.

3

Yes, the behavior is up to the linker itself. Note that another difference is that RPATH entries are inherited (used when loading libraries loaded by the current library) where as RUNPATH entries are only used when loading libraries referenced from the library with the RUNPATH entry.

4

I was wondering about that when trying to figure out if the order the needed share libraries show up in the ELF’s dynamic section mattered.

When dealing with C extensions for high-level languages, I believe a SO can only be loaded into the interpreter’s environment once (i.e. can’t have multiple definitions of the same symbols) and the C extensions are SOs that get dynamically loaded when first needed. As such, in this specific case, I don’t think there’s anything I can do to my extensions RPATH/RUNPATH to affect the version of an SO both my extension and the interpreter depends on because the linker can’t know that my extension is going to be needed at a later time. (But my understanding of all this is limited so that might be wrong.)

So I think my only option was to get my extension to work with the libraries the intepreter uses. (Which I did manage eventually.)

5

For ELF platforms, there is a global symbol table[1]. Once a symbol is bound by a library that is it (this is how LD_PRELOAD works. There’s also a flag (-Wl,-z,now?) for eager bindings that can’t be preloaded over. So if you load libstdc++ from GCC 4.8, loading a libstdc++ from GCC 13 later will add the set difference of symbols to the symbol table. Hopefully libstdc++'s ABI stability is accurate because you’re going to have GCC 13 code calling GCC 4.8 symbols where they exist already.

Note that MATLAB has also caused me pain years ago because they shipped an ancient Boost build that conflicted with our Boost build. Luckily, #define boost myboost in strategic spots can avoid this specific one, but their ABI cleanliness is quite annoying.

[1] There are symbol namespaces via dlmopen, but they are a very limited resource (16 by default), and glibc-only. Ends up being a lot of platform specific code for not a lot of gain.

1 Like
6

Well, that’s good to know! The fact that it’s not documented is a problem, so I’m glad I found this thread.

7

Note that CMake only models “rpath” as a concept (arguably CMake should have been named RUNTIME_LIBRARY_SEARCH_PATHS or something, but that ship sailed long ago); linkers started defaulting to DT_RUNPATH rather than DT_RPATH ELF entries years ago (all with the same flags).

2 Likes
8

I think the way CMake handles it is appropriate but it might be worth adding a note to the RPATH wike page I linked above. I’ve seen that page reference in several places discussing RPATH vs RUNPATH since it does a good job differentiating between RPATH and RUNPATH. But then it starts using the word RPATH everywhere. It would be helpful to mention in the “CMake and the RPATH” section that whether RPATH or RUNPATH is used is determined by the linker.

9

I agree. If someone wants to make that documentation page, I can help review it.

10

It needs to be made by someone who actually knows how this part of CMake works. Is there likely to be such a person outside Kitware?

11

LWN has published an article on how library loading works:

Though it kind of glosses over the library search procedure (though I’ve found man 8 ld-linux to be very useful for that).

12

For documentation, I have found that getting at least a pass over it by a non-expert helps to keep it useful for non-experts. For something like this, I think that describing how it works at all first gives the proper context for the knobs CMake offers abstractions for controlling it (with other things left to platform-specific flags which might be documented for the common use cases).

13

I’ll be the first to volunteer to make such an editorial pass. I volunteer!

14

Sure, but it’s not CMake’s job to document that. What’s missing from CMake’s documentation is simply what CMake does with these variable settings.

15

I agree; links to these resources may be useful rather than restated in CMake’s documentation. Note that library search is highly relevant to CMake and background is probably far more useful than “how symbols get located at an address”. Things like “why does CMake warn about not being able to make a reliable RPATH ordering?” truly need the background context to really make sense.

16

SGTM. Also there should be some description of the high-level uniform programmer experience CMake is trying to create by handling these different platforms in specific ways.

17

The last part of my “Deep CMake For Library Authors” deals specifically with RPATH and RUNPATH, including reference to the equivalent on macOS.