How to prevent CMake from adding host system flags (-arch arm64) when cross-compiling with ExternalProject_Add?

I solved it by properly passing the toolchain that I was using to build my main project. For future reference this is just by passing

CMAKE_ARGS
    "-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}"
    # ... rest of the args`