Here is an example of mine, with explanatory comments. Disclaimer: haven’t used this in production at this time, but it seems to be working fine on Linux and MacOS.
Note that there is also RUNPATH, which is used only if RPATH and LD_LIBRARY_PATH don’t contain the sought library. According to here it’s up to the linker to choose between using RPATH vs RUNPATH.