You’re overwriting variables all over the place here. You code is the same as:
set(CMAKE_SKIP_RPATH TRUE CACHE BOOL "If set, runtime paths are not added when using shared libraries.")
set(CMAKE_BUILD_RPATH_USE_ORIGIN TRUE)
set(CMAKE_SKIP_BUILD_RPATH FALSE)
set(CMAKE_BUILD_WITH_INSTALL_RPATH OFF)
set(CMAKE_VERBOSE_MAKEFILE ON)
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
${CMAKE_INSTALL_PREFIX} is almost always going to be an absolute path, so that’s one possible location for the absolute path to come from, though your settings seem to not be using that so much. I would suggest setting CMAKE_BUILD_WITH_INSTALL_RPATH to ON, making the build tree layout look like your install tree and using an install rpath of $ORIGIN/../lib (or something like it).