17.3 Undefined Behavior and Sanitizer-Style Tooling

Language semantics must distinguish several kinds of freedom. Defined behavior has a specified result. Implementation-defined behavior lets an implementation choose and document one result, such as a plain integer width. Unspecified behavior permits one of several results without requiring documentation. A defined trap deliberately stops execution. Undefined behavior (UB) imposes no requirements after the operation occurs.

UB is not a random value chosen at runtime. It means a conforming compiler may reason only about executions that avoid UB. If signed overflow is undefined, an optimizer can infer that x + 1 > x for every defined execution and simplify a branch. If the source overflows, observing wraparound in a debug build never created a language guarantee. Common examples include invalid pointer dereferences, out-of-range shifts, use of uninitialized values, violating alias rules, and accessing an object after its lifetime.

This contract enables optimizations but makes local reasoning dangerous. A check written after an overflowing operation may already be too late. Safe languages often replace UB with checked traps, wrapping operations, or restricted unsafe regions. Compiler IR must model its own semantics precisely: values such as poison or undefined placeholders are not interchangeable with arbitrary source values, and transformations must preserve when effects become observable.

Consider if (x + 1 < x) report_overflow();. Under wrapping arithmetic the condition is true when x is the maximum integer. Under checked arithmetic, evaluating the addition traps before the comparison. Under C-like undefined signed overflow, an optimizer may conclude that the condition is false for every execution it must preserve. The same surface syntax therefore supports three different legal transformations. Frontends must encode the selected contract accurately; adding a no-overflow IR flag merely because it improves code is a compiler bug.

Sanitizers make selected violations observable

Sanitizer-style tooling instruments a program during compilation and links a runtime that reports failures. AddressSanitizer (ASan) surrounds allocations with red zones and uses compact shadow memory to mark addressability. It detects many out-of-bounds and use-after-free accesses; a quarantine delays reuse so stale pointers remain detectable. It does not detect every temporal bug, and custom allocators or uninstrumented code can reduce visibility.

UndefinedBehaviorSanitizer (UBSan) inserts checks before operations such as invalid shifts, misaligned accesses, and selected overflows. Checks may recover and continue, report once, or trap; the chosen mode changes both evidence and runtime risk. MemorySanitizer (MSan) tracks whether bits are initialized and propagates that state through computations, requiring nearly all participating code to be instrumented. ThreadSanitizer (TSan) observes memory accesses and synchronization to identify data races using a happens-before model. Its cost and runtime design usually require a separate build.

Design the campaign, not only the flags

One overloaded sanitizer binary is rarely ideal. Maintain a matrix: fast ASan+selected UBSan tests on every change, slower MSan and TSan jobs on compatible platforms, and periodic broad fuzz campaigns. Instrument dependencies where possible, preserve frame pointers or equivalent unwind information, and retain debug symbols and exact build IDs. A report without symbolization, command line, input, and revision is much harder to act on.

Sanitizer overhead changes scheduling and memory layout. A race may disappear under ASan; an allocation bug may appear only with optimization. Test multiple optimization levels and reproduce findings in the smallest appropriate configuration. Suppressions should name an owner, justification, and expiry; broad suppression patterns silently destroy coverage.

Sanitizers demonstrate violations on executed paths. A clean run is not proof that the program contains no UB, memory error, or race. Combine them with static analysis, type and ownership rules, IR verifiers, fuzzing, review, and sandboxing. Also secure the tooling itself: a compiler processing untrusted source should have time and memory limits, because even a correctly reported violation can be paired with denial-of-service input. The honest claim is precise: which checker ran, over which corpus and configurations, and what it observed.