You can add the definition BOOST_ALL_NO_LIB to disable boost’s auto-linking. See the boost docs here: https://www.boost.org/doc/libs/1_81_0/libs/config/doc/html/index.html and do a search for BOOST_ALL_NO_LIB for some more info.
For others who are using CMake’s FindBoost module, you can achieve the same thing by adding the target Boost::disable_autolinking to your target link libraries. See https://cmake.org/cmake/help/latest/module/FindBoost.html for details.