Is it a module delivered as part of CMake? If yes, please give the information.
If no, how is consumed this library? through a imported target or not?
Did you check that these flags are not defined elsewhere (globally or at directory level using command add_compile_options or variable CMAKE_<LANG>_FLAGS)?
Currently, without a complete environment description, it is impossible to identify where come from your problem.