This is not supported. Build trees have absolute paths baked into them and are not transferable between (non-identical) machines.
The way to fix this is to have CMake use your target compiler via a toolchain file. This toolchain must run on the host you configure CMake on. See these docs for making a toolchain file for your compiler set.
Cross compiling is supported in CMake and used all the time. It may not support the specific method of using them in the way you describe here, but they are supported.