I removed the environment variables CFLAGS and LDFLAGS to avoid them being picked up by cmake.
This cmake behavior is quite problematic and undesirable. In my case a build machine had these environment variables defined giving a different build result. This kind of behavior is not good for reproducible builds and can be hard to track down. A toolchain file should be the input to the compiler and linker flags.