Can you try using cmake --trace-expand to see where that libm.dylib path is coming from?
cmake --trace-expand
libm.dylib