cmake picks up LDFLAGS from environment

CMake initializes the flags in the project() call (which enables the C and CXX language toolchains), so your setting is too late in your example.