It seems like CMake uses CMAKE_CXX_FLAGS_RELEASE and CMAKE_CXX_FLAGS_DEBUG (depending on the current value of CMAKE_BUILD_TYPE) in addition to CMAKE_CXX_FLAGS. The -O3 you see is likely set by default in CMAKE_CXX_FLAGS_RELEASE.
If you want to use only the flags you manually specified, you can set the build type-specific flags to the empty string like this:
SET( CMAKE_CXX_FLAGS_DEBUG "")
SET( CMAKE_CXX_FLAGS_RELEASE "")
SET( CMAKE_CXX_FLAGS "-Ofast -DNDEBUG -std=c++20 -march=native -fpic -ftree-vectorize")