1.1 What a Compiler Does

A compiler is not just a program that "turns code into machine code." That phrase is directionally true, but it hides the real engineering structure. A compiler is a sequence of analyses and transformations that tries to preserve the meaning of a program while changing its representation. At the beginning, the program is text written by a human. At the end, it may be machine code, bytecode, an object file, an intermediate artifact for another compiler, or even another high-level language. Between those points, the compiler repeatedly asks: what structure is present, what rules must hold, what information can be proven, and what representation is most useful for the next phase?

This course treats a compiler as a layered system. Each layer has a contract. The lexer does not need to understand function calls; it groups characters into tokens. The parser does not need to know target registers; it recognizes grammatical structure. The resolver connects uses of names to declarations. The type checker proves that operations are used according to the language's type rules. The IR generator lowers rich syntax into a simpler representation. The optimizer rewrites that representation while preserving observable behavior. The backend maps the representation onto an execution model.

That layered view is important because compiler failures are rarely random. A syntax error belongs to a different layer than an unresolved name. A type mismatch belongs to a different layer than an undefined external symbol at link time. A wrong optimization is more dangerous than a rejected program because it can silently change behavior. A real compiler is therefore a collection of small proof obligations, not a single magical translation pass.

The Core Pipeline

The classic teaching pipeline is:

source text
  -> lexer
  -> token stream
  -> parser
  -> syntax tree / AST
  -> resolver and type checker
  -> checked AST
  -> IR generation
  -> intermediate representation
  -> optimization
  -> optimized IR
  -> code generation
  -> target code

Real compilers may split, merge, repeat, or skip some of these phases. C and C++ have preprocessing before parsing. Java compiles source into JVM bytecode. JavaScript engines often parse, interpret, profile, compile hot code, deoptimize, and recompile. Rust and Swift do extensive semantic checking before code generation. LLVM-based compilers often lower from a language-specific AST into LLVM IR and then reuse a large middle-end and backend. The phase names vary, but the architecture keeps returning to the same idea: each stage turns one representation into another representation with more explicit structure or fewer source-language conveniences.

Think of each stage as a checkpoint. If a lexer accepts an invalid character silently, every later stage inherits confusion. If a parser builds an incorrect tree, the type checker may report misleading errors. If the type checker allows an impossible operation, the backend may be forced to generate code for a situation the runtime cannot support. Robust compilers make invalid states difficult to represent.

Representations Matter

The most important compiler design question is often not "which algorithm should I use?" but "what representation should this phase consume and produce?" A token stream is good for grammar recognition but terrible for type checking. An AST is good for user-facing diagnostics because it still resembles the source program. A control-flow graph is good for optimization because branches and loops become explicit. Static single assignment form is good for data-flow reasoning because each variable definition has a clean identity. Machine instructions are good for scheduling, register allocation, and ABI compliance, but they are too low-level for source-language type rules.

This is why compilers are full of intermediate forms. A beginner might try to parse directly into assembly, but that quickly becomes unmaintainable. Suppose the language later adds if, function calls, local variables, closures, or objects. A direct parser-to-assembly design forces every syntax feature to know too much about the target machine. A better design lets the frontend describe what the program means, then lets later phases decide how that meaning should be implemented.

Representations also control diagnostics. If the parser discards source ranges, the type checker cannot point to precise code. If the resolver does not record where a symbol was declared, an error cannot say "the previous declaration was here." If the optimizer loses mapping information, a debugger cannot explain optimized code in terms of source variables. Compiler architecture is therefore also user experience architecture.

Frontend, Middle-End, Backend

Compiler engineers often divide the system into three large regions:

• The frontend understands the source language.
• The middle-end analyzes and optimizes a language-independent representation.
• The backend understands the target execution environment.

The frontend is where syntax, declarations, modules, overload resolution, type checking, borrow checking, pattern checking, and most user diagnostics live. Its job is to reject programs that are not valid in the source language and to produce a checked representation for valid programs.

The middle-end is where many optimizations live: constant folding, dead code elimination, common subexpression elimination, inlining, loop optimizations, escape analysis, and data-flow analyses. The middle-end must be conservative. It may replace 2 + 3 with 5, but it must not replace a function call with a constant unless it can prove the call has no relevant side effects and always returns that value.

The backend is where target-specific details dominate: instruction selection, register allocation, stack frame layout, calling conventions, relocations, debug information, and object-file emission. A backend for x86-64 has different constraints from a backend for ARM64 or WebAssembly. Even when two targets support similar operations, their registers, instruction encodings, ABI rules, and memory models differ.

Diagnostics Are Part of the Compiler

Advanced compiler work is not only about accepting correct programs. It is also about rejecting incorrect programs in a way that helps users repair them. A compiler that says error is technically rejecting the program, but it is not doing enough engineering work. A good diagnostic should usually include a stable error code, severity, a primary source span, a concise message, secondary labels, and possibly a fix-it hint.

For example, consider:

let ok: bool = 42;

A weak diagnostic says:

type error

A useful diagnostic says:

error[E0301]: cannot assign int to bool
  --> main.mini:1:16
   |
 1 | let ok: bool = 42;
   |         ----   ^^ this expression has type int
   |         |
   |         variable declared as bool here
help: change the variable type to int, or compare the number to produce bool

That message depends on earlier architecture decisions. The lexer and parser must keep source locations. The type checker must know both the expected type and the actual type. The diagnostic layer must support labels, related spans, and suggestions. The compiler's internal representation and its user interface are connected.

A Single Program Through the Pipeline

Trace let total: int = price * count + 3; rather than treating the pipeline as a list of labels. The lexer emits LET IDENT COLON IDENT EQUAL IDENT STAR IDENT PLUS INT SEMICOLON, preserving each token's spelling and source span. The parser groups the right side as +( *(price, count), 3 ), not as a flat token list. Resolution binds price and count to declarations; type checking proves both operands of * are numeric and that the final value fits int. Lowering may create temporaries such as %0 = load price, %1 = load count, %2 = mul %0, %1, %3 = add %2, 3. Only much later does the backend decide whether %2 lives in a register, stack slot, or is folded away.

That trace explains a practical debugging rule: inspect the earliest representation that can contain the bug. A missing semicolon is not a type-checker problem. A wrong parenthesization is not a register-allocation problem. A wrong answer only under optimization should be reduced to IR before blaming source semantics. Compiler teams routinely dump tokens, ASTs, bindings, types, IR, and assembly for exactly this reason.

Engineering Checklist

• Give every cross-phase value an explicit invariant: for example, every token has a half-open source range, every resolved name has a declaration or a diagnostic, and every typed expression has either a valid type or an error type.

• Keep diagnostic construction separate from detection. The lexer/parser/type checker should report structured facts; one diagnostic layer should render messages, colors, JSON, IDE ranges, and fix-its.

• Preserve observability. A compiler that is fast but cannot print an AST or explain a failed optimization becomes extremely expensive to maintain.