For Linux, use and RPATH setting of $ORIGIN/rel/path/to/libdir should make that work.
$ORIGIN/rel/path/to/libdir