I would make a build tree per compiler instead of starting from scratch each time.
add_library(mytarget) # or add_executable()
target_compile_features(mytarget PRIVATE cxx_std_14) # or PUBLIC if you use C++14 in headers.
CMake’s downloads do not require special permissions to install; one extracts the tarball and uses it in-place.