3.3 Handwritten Lexers and Table-Driven Lexers

Once token rules are specified, we still have to implement the scanner. Two broad implementation styles appear repeatedly: handwritten lexers and table-driven/generated lexers. Both can be excellent. The right choice depends on language complexity, diagnostic requirements, team preferences, performance constraints, and tooling goals.

A handwritten lexer is normal code that calls helper functions such as peek, advance, match, identifier, number, and string. A table-driven lexer represents the DFA as data. It classifies each character, looks up the next state, records accepting states, and emits the longest token. Lexer generators sit between those ideas: the developer writes regex rules, and the tool emits tables or source code.

Handwritten Lexers

Handwritten scanners are popular in compilers that need customized diagnostics or unusual lexical rules. They are easy to debug with a normal step-through debugger because the control flow is source code. They can produce targeted messages: "numeric literal requires digits after ." is easier to write when the scanner knows exactly which branch failed.

A typical handwritten scanner loop looks like this:

scanToken():
  skipWhitespaceAndComments()
  start = position
  c = advance()
  if isLetter(c): return identifierOrKeyword(start)
  if isDigit(c): return number(start)
  switch c:
    case '=': return match('=') ? EQEQ : EQ
    case '"': return string(start)
    default:  return errorToken(start, "unexpected character")

The danger is drift. If token rules are scattered across many branches, the implementation can become harder to audit. You need systematic tests for every operator prefix, literal edge case, keyword conflict, and error recovery path.

Table-Driven Lexers

A table-driven lexer turns automaton execution into data lookup. The scanner keeps a state and classifies each character into a character class such as letter, digit, space, =, or other. Then it asks:

next = transition[currentState][charClass]

The scanner also remembers the last accepting state and position. This is how maximal munch works in a DFA. It may read one character beyond the last accepted lexeme, discover there is no valid continuation, then emit the token from the last accepting point and resume from there.

The advantage is regularity. The same compact loop can scan many token kinds. Generated lexers can compress transition tables, merge equivalent character classes, and minimize the automaton. The tradeoff is that custom diagnostics may be harder unless the generator supports semantic actions or error labels.

Generated Lexers

Lexer generators automate the path from token regexes to executable scanners. A generator can combine token NFAs, convert to a DFA, minimize or compress it, and emit code. This reduces hand-maintained scanner logic and keeps the implementation close to a formal token specification.

The generator is not a replacement for language design. You still must decide rule priority, skipped channels, keyword handling, Unicode policy, literal rules, error messages, and source-span conventions. You also need golden tests, because a concise regex specification can still encode the wrong language.

Choose the Implementation Shape by Change Rate

A handwritten lexer is often the clearest choice for a small language with context-sensitive lexical rules: indentation, nested comments, interpolation, or custom diagnostics can be expressed with direct loops and named states. Keep it disciplined: one cursor abstraction, explicit helper functions such as scanNumber and scanString, and no hidden cursor rewinds except a documented maximal-munch fallback.

A table-driven/generated lexer wins when many regular token patterns must remain consistent, when the grammar evolves through declarative rules, or when performance matters. The generated transition table is an implementation detail; wrap it in the same token/source-span API as a handwritten scanner so the parser cannot tell which implementation produced its stream.

Hybrid Is Normal

Many real lexers use a DFA for ordinary identifiers/numbers/operators and small handwritten modes for strings, interpolation, directives, or indentation. Measure before optimizing table layout. The most important property is agreement on a golden corpus: both implementations should emit identical kinds, payloads, spans, and diagnostics for every test file.