I have a complex multithreading program that, sadly, it is exhibiting random crashes for a couple of users on Windows. My program is a complex one, that compiles multiple static and dynamic libraries into, finally, an .exe all compiled with MSVC 2022 Community.
There are several third party libraries I don’t control, but I pass the CMAKE_BUILD_TYPE to them through an ExternalProject_Add. I have developed code to get a stack trace with files and line numbers and indeed that works fine on my development machine, where the .pdb files are indeed created.
I don’t mind opening my source code to the world, as it is an open source project, but my users have no idea on how to compile.
My problem is that the .pdb files are created in my staging area, and not embedded into the executable or into the .DLL, static .LIB files and they require the file system in the user’s machine to match mine to get files and line numbers in the stack trace. If I renamed my staging area (or it is not present) I don’t get files or line numbers in the backtrace.
I am wondering if there’s a way (CMAKE or MSVC flag) to embed the .pdb files directly into my libraries and executable (at least those built with cmake).
If you can require CMake 3.25 or later, CMAKE_MSVC_DEBUG_INFORMATION might be what you want. As the docs for that variable explain, all it’s really doing is selecting -Z7 (embed debugging information directly in the object files) instead of -Zi (embed debugging information in a separate .pdb file). Setting CMAKE_MSVC_DEBUG_INFORMATION to Embedded may be what you’re after.
Thank you Craig!
Actually, that’s not what I want. That makes it worse.
It embedded the .pdbs, but it still requires all the source code to be available and in the same location as in my machine. With .pdbs, I don’t need the source code. Just need to mimic the file layout.
What I am really looking is for a solution that creates a “fat” .exe with with all the file information embedded, like Linux dwarfs can do I believe, so that I don’t have to give neither the source nor have my users mimic the file layout.
I solved it. I left the default behavior of CMake’s RelWithDebInfo (ie. create all .pdb files), then used:
#include <dbghelp.h>
void printStackTrace()
{
...init code...
const std::string pdb_dir = mrv::rootpath() + "/debug";
if (!::SymSetSearchPath(hProcess, pdb_dir.c_str()))
{
// Handle error
DWORD error = GetLastError();
std::cerr << "SymSetSearchPath returned error: " << error << std::endl;
file << "SymSetSearchPath returned error: " << error << std::endl;
file.close();
return;
}
}
and I used CMake’s pre-package / package scripts to move all .pdb files into the packaged debug directory.
Consider looking at a service like sentry.io / bugsplat.com / others exist where you can use either Crashpad/Breakpad or their own custom wrapper libraries to catch crashes and upload a minidump to a service (self-hosted or cloud hosted). If you then upload your PDBs/symbols for your public releases to the logging site, you’ll get error reports with full stack traces… I’ve used sentry for a commercial product, I know that bugsplat has a free tier however.
Hi, what you intend – at least part of it – is completely unconventional.
Yes, you can use /Z7 as pointed out. This embeds the old CodeView-format debug streams in the COFF files. Maybe you could even coerce link.exe into embedding it inside the final PE files (but I think that scenario is no longer supported by Microsoft).
The CodeView (CV) format was abandoned (as in: not getting improved) at least two decades ago and PDB is the way to go. There is even a PDB format these days which is cross-platform (related, but not identical as far as I gather). And as far as my understanding goes not everything that can be represented in PDB can be represented in CV. Not sure if this is limited to the modern cross-platform PDBs only, though.
For static libs, however, I agree that /Z7 would be best practice. Why? Because static libs are merely archives of object (COFF) files. And the final linking stage into a PE file (i.e. .dll or .exe) is going to place the debug information into a PDB file anyway (if requested). But by using /Z7 you are going to remove the burden of having to keep the PDB matching that .lib around.
Alas, there are a few pitfalls:
- the handling of debug symbols can slow down the overall build process considerably and I have run into issues on really big projects in the past
- you should use the full PDB for release builds and disable incremental linking
- for the static libs you should make sure that they don’t have undue external dependencies, e.g. I prefer to
/nodefaultlibout those C/C++ runtime libs when building static libs
Your problem statement suggests that you want to set up a symbol server. That server uses particular values from your PE file to match up the corresponding values. symstore is used to populate it. You could easily symstore to a local “staging area” which you then rsync over to some server. But Mozilla also has (or had) some Python scripts to work with the symbol servers in a way DIA/dbghelp expects. Then point your debug code to the symbol server. This may be a good starting point (or this one).
Additional info:
- use
/pdbaltpath:%_PDB%(linker command line) to remove the path leading up to your PDB file - make use of
LINK/_LINK_,LIB/_LIB_andCL/_CL_environment variables to smuggle command line options into builds over which you have limited control … later options always override previous ones- the ones without underscores come first, then comes whatever is on the command line and then come the options from the environment variables with the underscores
- consider using
/Brepro(all tools) to make your builds (more!) reproducible; unfortunately certain things are really hard to get done deterministically (but MS has been doing it for Windows [1] and only certain PE file parts [2] contribute to the symbols … i.e. one PDB for multiple DLLs is possible)
PS: in addition to the named tools I’d like to throw Google BreakPad and CrashPad into the discussion. You can totally achieve the same as those mentioned services with your own symbol server plus one of these (or similar alternatives). Sentry advertises integration with the BreakPad features, btw.
PPS: unfortunately the system limits me to only include two links per post. Which means I need to remove some and can hopefully post them in a followup … 
The two missing links, which I hope the system will allow to post:
- Introducing Winbindex - the Windows Binaries Index – m417z / blog – A Blog About Stuff
- Determinism Bugs, Part Two, Kernel32.dll | Random ASCII – tech blog of Bruce Dawson
None of these links are there to promote anything, so no idea why this limit … 
And to reiterate what I mentioned above: /Z7 would be unconventional these days and embedding of debug streams into the PE unsupported. On the Linux side with ELF and DWARF you are more flexible on one hand, but on the other it is also common to split the symbols off by way of objcopy or similar into a separate file and set objcopy --add-gnu-debuglink=$name.dbg $name to make the association between the two.
Also keep in mind that you may want to strip your private symbols for a public symbol server (Microsoft does that for theirs, notwithstanding the occasional slip-up over the years).
/Z7would be unconventional these days…
That depends on your situation. If you want to use a compiler cache like ccache, you can’t use PDB. You must use /Z7 for that, otherwise you essentially disable all caching. And the build performance gains from a compiler cache can be one or two orders of magnitude faster builds. For those who don’t need a PDB for local development and can use the embedded debug info, it’s a no-brainer to go with /Z7.
Totally agree with that assertion (assuming you meant – and /Z7 sort of implies that – the PDB generation that happens during compilation rather than during linking). But that amounts to the same as working with a static lib. In the end a PDB is generated by the linker, if requested (and no embedded debug streams exist with modern MSVC tooling).
There are quite a few points one has to consider with compiler caches indeed. But of course you’re right.
However, the way that cl.exe works conventionally these days (meaning how Microsoft evidently uses it with MSBuild etc), it’ll act as a server and multiplex the compilation to individual cl.exe processes. But it’s true, this is one point were cl.exe diverges from how other compilers work, because it allows you to give one output directory (instead of .obj path) and takes an (almost) arbitrary number of .cpp files to compile on a single command line (well, technically in response files). That aspect alone also requires some funny jumping through hoops because compiler caches don’t bother emulating that behavior and instead expect a 1:1 relationship of compiler input to output, given a particular command line.