1.4 Course Project: A Small Language and Compiler

This course will repeatedly return to one project: a small language compiler. The project is not meant to compete with C, Java, Rust, or Python. Its purpose is to make compiler theory concrete. A tiny language can still contain every essential compiler idea: tokens, grammar, AST, scopes, types, IR, control flow, optimization, runtime representation, code generation, testing, and diagnostics.

The danger in compiler projects is uncontrolled ambition. It is easy to say "let us add functions, closures, objects, generics, modules, macros, pattern matching, exceptions, async, ownership, and native code." Each feature sounds isolated. In reality, each feature touches multiple phases. Closures affect parsing, name resolution, type checking, IR, runtime allocation, calling convention, optimization, debugging, and garbage collection or ownership. Objects affect layout, dispatch, method lookup, access control, initialization, and runtime metadata. Generics affect type representation, overload resolution, monomorphization or reification, diagnostics, and code size. The advanced lesson is simple: language features are not syntax decorations; they are whole-system commitments.

The First Language

The first version of our language should be intentionally small. A useful starting point is:

fn main() -> int {
  let x: int = 40 + 2;
  return x;
}

This tiny example already requires:

• A lexer for keywords, identifiers, punctuation, operators, integer literals, and EOF.
• A parser for functions, blocks, variable declarations, returns, and expressions.
• An AST with source ranges.
• A symbol table for local variables.
• A type checker for int, return statements, and arithmetic.
• An IR that can represent constants, addition, local values, and return.
• A backend that can execute the IR, emit bytecode, or eventually produce target code.
• A test harness that can compare tokens, AST snapshots, diagnostics, IR, and final output.

The first project goal is not performance. It is an end-to-end vertical slice. A compiler that accepts one tiny program and produces a verifiable result is more valuable than a huge parser that cannot run anything.

Milestones and Interfaces

Compiler phases should be tested independently and together. The lexer can have golden token tests. The parser can have AST snapshot tests. The type checker can have positive and negative programs. The IR generator can compare textual IR. The optimizer can compare before/after IR and run equivalence tests. The backend can execute compiled programs and compare output or exit codes.

The interfaces between phases are as important as the phase algorithms. If parser nodes do not carry source spans, diagnostics suffer. If symbol bindings are stored as strings instead of stable symbol IDs, shadowing becomes fragile. If the typed AST mutates the untyped AST in place without discipline, tests become hard to reason about. If IR instructions do not clearly model control flow, optimization becomes unsafe. Treat every phase boundary as an API.

For chapter 1, the project is still architectural. You should be able to answer:

• What is the smallest program the compiler will run?
• What are the first token kinds?
• Which AST nodes are required for the first milestone?
• What type rules exist in version one?
• Will the first backend interpret IR, emit bytecode, emit C, emit WebAssembly, or emit native assembly?
• What will count as proof that the first milestone works?

A Practical Feature Budget

A good first language might include:

• int and bool
• arithmetic and comparison
• let
• if / else
• while
• functions with typed parameters
• return
• comments

It should probably not include, in the first version:

• classes
• closures
• generics
• exceptions
• modules
• operator overloading
• macros
• implicit conversions
• borrow checking
• concurrency

This is not because those features are uninteresting. They are extremely interesting. The issue is sequencing. A compiler course should let you feel the weight of each mechanism. If you add too many features before the pipeline exists, the project becomes a fog. If you add one feature at a time after the vertical slice works, each feature becomes a controlled experiment.

What to Keep in the Repository

From the first day, keep these artifacts versioned:

examples/
  ok/
  errors/
tests/
  lexer/
  parser/
  typecheck/
  ir/
  execution/
docs/
  language.md
  diagnostics.md

The examples/ok directory contains programs that should compile. The examples/errors directory contains programs that should fail with specific diagnostics. Lexer tests pin tokenization. Parser tests pin tree shape. Type-checking tests pin semantic rules. IR tests pin lowering decisions. Execution tests pin runtime behavior. Documentation prevents the language from becoming only whatever the current code happens to accept.

This course will gradually turn that skeleton into a working compiler.

Build Vertical Slices, Not a Tower of Stubs

Start with an end-to-end fragment such as integer literals, addition, and print. It should lex, parse, build an AST, evaluate or lower, and emit an observable result. Then add one feature at a time across every affected phase: names require declarations, a symbol table, diagnostics, and tests; conditionals require boolean semantics, control-flow representation, and code generation; functions require scopes, calls, frames, and ABI decisions. A vertical slice reveals missing contracts early, whereas completing every lexer rule before any parser exists hides integration failures.

Keep one executable reference semantics, even if it is a tiny tree-walk interpreter. Differential tests can compare its output against generated code for small random programs. This does not prove a compiler correct, but it catches astonishingly many lowering and optimization mistakes.

Project Discipline

• Freeze a tiny language specification before each milestone: grammar, typing rules, observable errors, and examples.

• Design error nodes and an error type early, so a malformed program does not cause dozens of secondary crashes.

• Make every milestone demoable with input, emitted representation, and expected output. Compiler work becomes much clearer when it can be inspected phase by phase.