Debugging

Start with a reproducible failure, then narrow down where the behaviour first goes wrong. Change one thing at a time.

Record and replay

rr records a supported Linux program and replays it deterministically. Reverse execution lets you work backwards from a corrupted value to the write that changed it.

Recording changes execution timing and runs application threads on one core, so it may not reproduce every concurrency bug.

Sanitizers

ASan detects many buffer overflows and use-after-free errors.
UBSan detects selected forms of undefined behaviour.
TSan detects data races in executed code.
MSan detects uses of uninitialised data and requires instrumented dependencies.

These tools do not prove a program is correct. ASan and TSan need separate builds. Valgrind can inspect many binaries without recompilation, but adds substantial overhead.

Core dumps and post-mortem

A core dump can preserve a crashed process's state. Keep the matching binary and debug symbols so the backtrace is useful. On systems using systemd-coredump, coredumpctl helps find and inspect dumps.

Bisection at every level

git bisect run searches for a breaking commit using a test script. Make the test reliable before starting.

The same approach works on inputs and configuration: remove parts until the failure disappears, then narrow the remaining case. Keep the smallest reproducer.

Attaching to something already running

A debugger can pause a process. For a live service, consider tracing tools such as strace, perf or bpftrace, and check their overhead and permissions first.

What the optimiser did to your program

Optimised code may step through source lines out of order or omit variables. A build with debug information and -Og can make stepping easier.

If a failure depends on optimisation, check for undefined behaviour before reporting a compiler bug. A report needs the reproducer, compiler version and build flags.

Write down the mismatch

Record the expected result, actual result, exact input and command used. Include relevant versions and configuration. Separate what you observed from what you think caused it.

Choose a hypothesis that predicts a visible result. Inspect the value at the boundary where it could first become wrong. If the prediction fails, update the hypothesis rather than adding unrelated changes.

Breakpoints and watchpoints

A breakpoint stops execution at a location. A conditional breakpoint stops only when its condition holds. A watchpoint stops when a watched memory location is accessed or changed, depending on debugger support.

Inspect the call stack, local variables and arguments together. A correct value in the wrong invocation can be misleading. Hardware watchpoints are limited, and optimised variables may not have a stable memory location.

Logs that help

Record the operation, relevant identifiers and the transition that failed. Correlation IDs connect events from the same request across components. Include timing when ordering matters.

After fixing the cause, rerun the original reproducer and the nearby checks. Remove temporary instrumentation that adds noise or exposes data unnecessarily. See Testing for turning a failure into a regression test.

Reading