CMake is doing what you are telling it to do. The installed directory structure is independent of the directory structure used in the build directory. In your case, you have let install(TARGETS) use the default install destinations, which is appropriate when only installing a single configuration. However, as you’ve discovered, if you install multiple configurations, the last one will overwrite the earlier installed configuration’s binaries.
You need decide whether you want to install binaries to the same directory but with different file names, or with the same file names but installed to different directories. Either way, you will need to specify additional details compared to what you have now. Of the two strategies, personally I’d probably go with same directory, different file names. There is precedent for that with libraries, with debug libraries often having a d or _debug suffix on the library’s base name. On linux, this would give you files like libblah.a for Release and libblahd.a or libblah_debug.a for Debug. You would set these most easily with the OUTPUT_NAME or OUTPUT_NAME_<CONFIG> target properties. This approach also has the advantage that it doesn’t change the relative paths to things, so embedded RPATH details will be much easier to specify.