Isn’t add_dependencies causing that behavior? Did you try to create a separate target for the third party lib (so it compiles with default flags) and then link it with your main lib using target_link_libraries?
Is it a module delivered as part of CMake? If yes, please give the information.
If no, how is consumed this library? through a imported target or not?
Did you check that these flags are not defined elsewhere (globally or at directory level using command add_compile_options or variable CMAKE_<LANG>_FLAGS)?
Currently, without a complete environment description, it is impossible to identify where come from your problem.