-
Notifications
You must be signed in to change notification settings - Fork 73
Debugging Native Applications
One of the highest impact things cargo-dist can do is help users optimize the crap out of their final binaries while still making them debuggable locally and remotely. Situations that might merit debugging include:
- Uncontrolled Crashes (segfault, abort, ...)
- Controlled Crashes (panic, unwind, ...)
- Inspection (gdb, lldb, rr, windbg, ...)
- Profiling (perf, instruments, ...)
All of these things have wildly different tools/implementations but they're all driven by the same kinds of metadata, variously referred to as "debuginfo", "symbols", or "sourcemaps". This info is stored in the final binary (.exe, .dll, ...), intermediate build files (.o, .a, .rlib, ...), and/or dedicated symbol files (.pdb, .dSYM, .dwo/.dwp, .map, .sym, ...).
Storing info in the binary is really great for making debugging simple and reliable, and even enables the binary to introspect its own execution. This is in fact mandatory for the normal behaviour of unwinding/backtraces (panics). It however has three downsides:
- You may not have the binary anymore, as is often the case when processing remote/saved results from profiling or crashreporting
- It makes the binary bigger (like, 10 or 100 times bigger if you include everything!)
- It fills the binary with a bunch of really rad information you might not want end-users to have so easily (not my concern, but some people care about it)
Hopefully it's obvious that storing debuginfo in intermediate files is simply a transient convenience for debugging during development, and not particularly useful for production environments.
Storing info in symbol files is in principle the gold standard that everyone should aspire to, because it keeps the binary you ship nice and lean but makes it possible to fetch all the info you need when it's time to debug -- even if debugging on a different machine! The downside is that you now need Infrastructure to store/fetch these symbol files, and you need to properly configure tools to use that infrastructure. Doing a good job of this across every platform is a frankly enormous task, which is why I usually tell people to Just Give Sentry Money To Do It For You. But even if you're doing that you still need to setup your builds properly, and oops cargo-dist is your builds so I need to get this right and tell you the various tradeoffs and how to work with them.
I think on balance the practical solution (if you're not hardcore about binary size or obfuscation) is to have a split-debuginfo file containing the bulk of the information, but to leave some small amounts of info in the binary to make things work ok when that file is missing but the binary is available. To understand what that might look like, we need to dig into what the available information is.
Basically anything! Binaries are generally container formats (ELF, Mach-O, PE32+) with some kind of index pointing to arbitrary sections (often DWARF). This means you can generally come up with your own custom sections and shove them in there and anyone who doesn't know about it can ignore it. People come up with custom instrumentation schemes that require custom metadata all the time. Even the "standard" formats are adding "official" sections/features all the time. That said, there's two really big and important categories:
- Unwind Tables (usually always in the binary, often regarded as distinctive from "debuginfo" or "symbols")
- Source Info (people usually just call this "debuginfo" or "symbols")
These are generally enormous (compressed) tables that map every address in the binary to facts about that address.
In this case of unwind tables the table stores "an entire program to execute in the unwinder's VM to unwind from this instruction byte in the program". How to restore caller-saved registers, how to run destructors, how to stop the unwind, and so on. Needless to say, storing an entire subprogram for every byte in the "real" program takes up a lot of space, even if you compress it really well! Of course unless you compile with panic=abort you probably need those tables in the binary to properly run it! These tables are also incredibly important to crashreporters and profilers, which both need to get backtraces for arbitrary addresses in the program to figure out what functions are currently executing. This is why I Am Always Screaming That You Should Build With Framepointers Enabled To Make Things Work Better.
In the case of source info, the tables can store:
- What source function each address of the binary is part of
- What file that function was from
- What line (and column/span) in that file it was
- The name of the function (usually mangled?)
- Where the values of variables are stored in memory (and how to read them?)
You might be thinking "hey aren't these ambiguous in a fully optimized binary?" and you would be absolutely right! If you run an optimized binary in gdb, you're likely to see "optimized out" a lot. But hey, you can make an effort! Also the optimizer can try to play nice and avoid optimizations that really fuck things up like Tail Call Optimization (which "break" backtraces) or Merging Identical Functions (which makes it ambiguous which function you're actually running).
One of the nastiest optimizations for debugging is actually inlining. Suddenly, one byte in the program is Genuinely Inside 7 Functions At Once. You can track this info -- there's tables for inlinee-info -- but empirically those are some of the most enormous tables to store, and as such are some of the easiest to put on the cutting room floor. Major tools like breakpad were discarding this info until about 2021. When we deployed them our symbol files got bigger by an order of magnitude!
A "balanced" amount of information to include in a binary would be unwind tables and function names. I believe this is the default on windows and macos even for optimized binaries with panic=abort, while linux opts for more brutal defaults.
TODO: flesh this out
- cargo profiles (debug=2, split-debuginfo, panic=abort, strip, ...)
- rustc flags
- tools (strip, lipo, dump_syms..)
- fuckin'... linker scripts..?
TODO: flesh this out
- dwarfdump
- dump_syms
- lldb, gdb, windbg, ...
TODO: flesh this out
General Formats:
- PE32+ (Windows)
- ELF (Linux)
- DWARF (Linux/Apple)
- Mach-O (Apple)
Symbol Files:
- pdb (Windows)
- dSYM (Apple)
- dwo/dwp (Linux)
- map (WASM/JS)
- that anti-binary thing you get from stripping a Linux binary?
- Breakpad Symbol Files (Universal but not supported by most things)
TODO: flesh this out
- debugger (windbg, visual studio)
- profiler (wpa)
- crashes (minidumps, breakpad-minidumps)
- debugger (gdb, lldb, rr)
- profiler (perf, samply)
- crashes (coredumps?, breakpad-minidumps)
- debugger (lldb, xcode)
- profiler (instruments, samply)
- crashes (?, breakpad-minidumps)