17.1 Unit, Golden, Snapshot, and Integration Tests
A compiler test suite must answer more than “did the process exit successfully?” A compiler is a pipeline of contracts: the lexer preserves token boundaries, the parser builds the intended tree, name resolution binds the right declaration, optimizations preserve observable behavior, and code generation obeys the target ABI. A useful suite places tests at several boundaries so that a failure is both detectable and diagnosable.
Unit tests isolate a small deterministic component. A scanner test may map source text to token kinds and byte spans; an IR verifier test may feed one malformed block and expect a precise invariant violation. Unit tests are fast and localize faults well, but mocks can hide broken stage boundaries. Prefer real value objects and small fixtures over elaborate mock graphs. For pure transformations, property tests add broader claims: pretty-printing then parsing should preserve structure, alpha-renaming should not change behavior, and every emitted branch target should exist.
Integration tests cross real boundaries. They can compile a source file, assemble and link it, run the executable in a controlled environment, and compare its output and exit status. They catch disagreements that isolated tests miss, such as debug locations using byte offsets while the renderer expects display columns. They cost more and produce larger failure surfaces, so retain intermediate artifacts and stage logs.
Golden files are reviewed contracts, not automatic truth
A golden test stores a trusted output—tokens, AST text, diagnostics, assembly, or object metadata—and compares a new run against it. A snapshot is the same broad idea, usually managed by a testing framework and updated conveniently. These tests are excellent when output is rich and a hand-written assertion would omit important detail. One diagnostic fixture can preserve severity, code, source ranges, notes, and fix-its at once.
The danger is the “update all snapshots” habit. Regenerating expected output with the compiler under test can approve the bug that changed it. A golden difference requires classification: semantic changes alter meaning or executable behavior; representational changes alter harmless formatting; nondeterministic changes expose unstable iteration order, paths, addresses, locale, or timestamps. Reviewers should understand which category applies before accepting a baseline.
Normalize only fields whose variability is explicitly irrelevant. Replace a temporary root with <TMP>, sort a set whose order has no contract, and pin locale and target. Do not erase instruction order merely because a failing optimizer test becomes green afterward. It is often better to compare a structured representation than fragile text: parse JSON diagnostics and assert fields, or disassemble an object and ignore section addresses while preserving relocations.
Build a portfolio with distinct failure modes
End-to-end tests provide confidence that the product works, but they should not replace narrow tests. A balanced portfolio uses many cheap unit and invariant tests, focused golden tests for rich surfaces, component integrations at risky seams, and a smaller cross-platform end-to-end matrix. Regression tests for every fixed compiler bug should contain the smallest reproducer and the expected failure mode, not merely a copy of a large customer project.
Mutation testing is a useful audit: deliberately invert a condition, remove an optimization precondition, or corrupt a relocation kind. If no test fails, the suite does not defend that behavior. Negative tests matter equally; invalid input should fail with the correct diagnostic without crashing or silently emitting code.
Keep fixtures hermetic. Pin tool versions, clear inherited environment variables, control clocks and random seeds, and isolate file systems. Record enough context to reproduce a failure: compiler build ID, flags, target triple, dependency versions, and seed. Parallel sharding should be deterministic, while flaky tests should be investigated and quarantined with ownership and an expiry—not silently retried forever. The goal is not the largest test count; it is fast, reviewable evidence covering independent risks.