I have built the googletest lib on windows with debug mode.
It will generate gtestd.lib and gtest_maind.lib.
But find_package(GTest) will fail to find them because of the suffix “d”.
I think the problem lies in the FindGTest.cmake:L205.
The FindGTest module will need to be updated to find the debug library variants. It appears as though FindXercesC.cmake does the proper logic that could be used as a template to get FindGTest.cmake to do the right thing here. Feel free to submit a merge request or an issue to get it fixed in CMake itself.
I’ve found that absorbing gtest directly into your build is a more robust and easier to use approach. It gets built with the same flags and settings as the rest of your project, which inherently takes care of most of the issues like the one you are facing. I recommend you take a look at the FetchContent module, the documentation for which even has examples for gtest.