17.2 Differential Testing and Fuzzing

Compiler bugs often hide in programs no engineer thought to write. Differential testing and fuzzing generate many such programs, but generation alone is not enough: a campaign needs an oracle, a rule that decides whether an observed result is suspicious.

A reference interpreter is a strong oracle when it implements the language simply and independently. Run the same defined program through the interpreter and optimized compiler, then compare observable output, exit status, and permitted side effects. Cross-compiler testing instead compares two or more implementations. Agreement is evidence, not proof—implementations can share a bug or make different legal choices. Metamorphic testing avoids requiring a known answer. It transforms a program in a way that should preserve a relation: rename locals, add a dead binding, reorder independent declarations, or compile at two optimization levels. A violated relation identifies a candidate bug.

The harness must filter programs with undefined or implementation-dependent behavior. Otherwise x / 0, signed overflow, unspecified evaluation order, target-width differences, or uninitialized reads create legitimate disagreement. Normalize only permitted variation such as diagnostic paths, and compare at the correct boundary. Floating-point results may require a semantics-aware relation rather than text equality.

For example, a typed generator might create let x: i32 = 7; print((x * 3) - x);. The harness evaluates its AST with the reference interpreter, compiles it at -O0 and -O2, and requires all three executions to print 14. It stores each result separately instead of voting by majority: if both compilers share a lowering bug, two matching wrong answers must not overrule the independent reference. Timeouts, compiler crashes, runtime traps, and output mismatches are distinct outcomes with different owners.

A fuzzer is a feedback loop

Mutation-based fuzzers start from a corpus and edit bytes or tokens. They are easy to deploy and good at exercising parsers, but random bytes rarely reach deep optimizer paths. Generation-based fuzzers produce inputs from a grammar or typed AST; they can guarantee syntactic validity, scope correctness, and type correctness. Hybrid systems mutate trees, lower them to source, and occasionally splice corpus fragments.

Coverage guidance rewards inputs that discover new control-flow edges, comparisons, or data-flow features. Coverage is a navigation signal, not the objective itself: two inputs may visit the same edge while exposing different values, alias patterns, or IR shapes. Domain-specific feedback—new opcode combinations, optimization rules, type pairs, or verifier states—can drive a compiler fuzzer deeper.

The initial corpus should be small and diverse. Include each syntax form, important type, boundary constant, and known regression, then minimize files whose coverage is redundant. Too many large seeds slow every cycle. Run instrumented compiler builds with assertions and suitable sanitizers so memory corruption and invariant violations become visible immediately. Give each execution time, memory, and output limits; compilers process hostile input in editors and build services, so hangs and resource explosions are real security findings.

Turn crashes into durable evidence

A raw fuzz failure may contain thousands of irrelevant tokens. A reducer repeatedly removes or simplifies fragments while preserving an interestingness test: same sanitizer class and top stack frames, same wrong-code mismatch, or same compiler assertion. Merely preserving a nonzero exit code can drift from the original bug to a trivial parse error. Language-aware reduction must keep declarations, types, and control-flow constraints valid.

Deduplicate by a stable signature, but do not trust stack hashes alone; inlining and allocator behavior can merge or split issues. Store the minimized reproducer, seed, compiler revision, complete flags, target, environment, oracle outputs, and reduction predicate. Add the final case to the ordinary regression suite before closing the defect.

Fuzzing is statistical. Report executions per second, unique paths or domain features, corpus size, time-to-first finding, flaky rate, and untested configurations. Rotate seeds and build modes, but preserve reproducibility for every finding. A mature campaign continuously converts random discoveries into small, deterministic specifications of compiler behavior.