3.4 Lexer Errors, Source Positions, and Diagnostics

Lexical errors are often the user's first contact with a compiler's quality. A weak lexer says "invalid token." A useful lexer says what text is invalid, where it begins, why scanning failed, and how the user might repair it. Diagnostics are not a cosmetic layer on top of compilation; they are part of the compiler's contract.

The lexer is uniquely responsible for character-level locations. Later phases may know that a statement is malformed, but the lexer knows exactly where a string literal began, where a bad escape appeared, or where an unknown character interrupted scanning.

Source Positions and Spans

A source position usually includes offset, line, and column. An offset is convenient for tools because it indexes directly into the source buffer. A line and column are convenient for humans. A span is a pair of positions or offsets that covers a source range.

For this input:

let bad = @value;

a good lexical diagnostic might be:

error[LEX001]: unexpected character '@'
  --> example.lang:1:11
1 | let bad = @value;
  |           ^
  = hint: remove it or use a valid operator

The primary span should cover the smallest useful range. For an illegal single character, that is one character. For an unterminated string, the span might begin at the opening quote and extend to the newline or end of file. For an invalid escape, the span might cover only the escape sequence.

Unterminated Structures

Unterminated strings and comments are special because they affect synchronization. If the lexer sees:

let s = "hello
let x = 1;

it must decide whether the string error ends at the newline or consumes the rest of the file. Many languages do not allow ordinary strings to cross line boundaries, so the lexer reports the opening quote and recovers at the next line. Block comments are different:

/* open comment
let x = 1;

If block comments may span lines, an unterminated block comment often consumes through EOF. Reporting one clear diagnostic is better than producing dozens of fake parser errors for text that was inside a comment attempt.

Recovery Strategy

Recovery is the art of continuing after a real error without lying to the user. A lexer can skip one bad character, skip to a line boundary, synthesize a token, emit an error token, or stop scanning. Each option has costs.

Skipping one character preserves context, but it can create cascaded errors if the original problem was an unterminated string. Skipping to the next line resynchronizes quickly, but it loses the rest of the current statement. Synthesizing a token can help the parser continue, but the diagnostic must make it clear that the token was invented. Emitting error tokens gives the parser visibility into lexical failures, but it complicates grammar recovery.

The policy should be documented and tested. For a teaching language, a reasonable policy might be:

illegal character        -> report, skip one character
unterminated string      -> report, recover at newline
invalid escape           -> report, continue inside string
unterminated block comment -> report once, recover at EOF

Source Positions Need a Precise Model

Pick one internal coordinate system, usually byte offsets into UTF-8 source, and derive display line/column lazily from a line-start table. Columns are trickier than they look: a Unicode code point, a UTF-16 code unit, and a terminal display column can differ. IDE protocols may require UTF-16 positions while compiler files use byte offsets. Convert at boundaries, document the convention, and test non-ASCII identifiers and tabs.

For a malformed token, consume enough input to guarantee progress but do not swallow unrelated syntax. An invalid character can become one error token; an unterminated string should usually consume until newline or EOF while retaining the opening quote span. Report the primary error once, then let later phases see a well-defined error token or halt according to the product's recovery policy.

Diagnostic Quality Checks

• Point at the smallest useful range, then add a secondary label for the opening delimiter or related declaration.

• Never calculate columns by repeatedly scanning from file start; line maps make diagnostics linear in file size rather than quadratic in error count.

• Snapshot exact ranges and messages in tests. “An error occurred” is not a sufficient compiler regression test.