From the install(RUNTIME_DEPENDENCY_SET) documentation:
Targets built within the build tree will never be installed as runtime dependencies, nor will their own dependencies, unless the targets themselves are installed with install(TARGETS).
This is deliberate. CMake has its own procedure for installing targets that it builds. Installing them with install(RUNTIME_DEPENDENCY_SET) would bypass this procedure. This command is meant for bringing in libraries external to the CMake buildsystem (either imported targets or libraries linked the “dumb” way with a -l flag.)
You need to install mylib and dep explicitly.