Do not share a FETCHCONTENT_BASE_DIR between different builds. That shares more than just the downloaded sources, it also shares all the bookkeeping files used internally. Those do not expect to have the source directory changing between runs. In particular, the sub-build set up to do the download won’t tolerate the architecture changing, which is what you’re seeing because the sub-build uses the same generator as the main build.
While I understand the desire to want to avoid duplicating the downloaded content when you have two or more builds that need the same thing, that isn’t something that FetchContent supports very well right now.