15.3 JIT Compilation Pipeline

An interpreter starts quickly because it performs little compilation, but it repeatedly pays decode, dispatch, type-check, and generic-operation costs. A just-in-time (JIT) compiler observes a program while it runs and compiles selected code to native instructions. The key word is selected: compiling every function aggressively at first use can spend more time and memory compiling than the program would have spent interpreting.

Modern runtimes therefore use tiered execution. A function may begin in an interpreter, move to a fast baseline compiler after enough calls, and reach an optimizing compiler only after stronger evidence of sustained hotness. The runtime can move execution between tiers while preserving one language-level state.

Hotness and tiering turn compilation into an investment

Counters on function entries and loop backedges estimate how often code executes. When a counter reaches a threshold, the runtime queues compilation. A baseline compiler translates bytecode almost mechanically, produces machine code quickly, and may add lightweight profiling. An optimizing tier builds richer IR, uses runtime feedback, performs inlining and specialization, allocates registers, and emits metadata for GC, exceptions, debugging, and deoptimization.

Thresholds balance competing costs. A low threshold compiles more cold code, increasing startup time and code-cache pressure. A high threshold leaves genuinely hot code in slower tiers for too long. Compilation may run on background threads, but it still consumes CPU and must synchronize code installation safely. Mobile, interactive, server, and short-lived workloads need different policies.

One useful model compares the one-time compilation cost with per-iteration savings. If optimizing compilation costs 2 ms and saves 20 ns per iteration, roughly 100,000 future iterations are needed just to break even. Real systems also include baseline cost, queue delay, deoptimization, instruction-cache effects, and profile accuracy.

Speculation enables speed—and requires an exit

Dynamic languages permit an operation to see many types and object shapes, yet one hot site often observes a stable subset. The optimizing JIT can speculate: compile x + y as unboxed integer addition under guards that both values are small integers and overflow does not occur. A property load can assume receiver shape 17 and use a constant field offset. A call can inline the one target seen by profiling.

Each assumption needs a guard. When a guard fails, execution cannot simply continue through specialized machine code. It enters deoptimization, transferring to a generic tier at the equivalent language-level point. The JIT emits a deoptimization record at each safepoint or guard exit. That record explains how optimized registers, stack slots, constants, and eliminated objects reconstruct interpreter or baseline locals, operand stack, and bytecode position.

For example, scalar replacement may remove a short-lived Point allocation and keep fields only in registers. If a later shape guard fails, the deoptimizer may need to materialize a real Point object before resuming. Inlined functions need virtual frames so the reconstructed stack contains caller and inlined callee states. Recovery must preserve exceptions and side effects: resume after operations already committed, but before operations not yet executed.

Code cache, safepoints, and invalidation

JIT-generated code lives in a managed code cache. The runtime allocates writable memory for emission, applies relocations, flushes instruction caches where required, and changes pages to executable rather than leaving them writable and executable. Entry stubs or indirection cells let the runtime install new versions atomically while existing threads finish old code.

Safepoints provide locations where thread state is describable to the garbage collector and runtime. Stack maps identify live references in registers and slots. Polls let the VM request GC, invalidation, debugging, or interruption. A moving collector may patch code or use handles; embedded object pointers need relocation and lifetime management.

Assumptions can become invalid without a currently executing guard. Loading a subclass may invalidate a “no subclasses” optimization; redefining a method may invalidate inlined calls. Runtimes record dependencies from compiled code to assumptions, mark affected code non-entrant, redirect future entries, and deoptimize active frames when they reach safe transfer points.

JIT correctness needs differential and stress testing across tier transitions. Run with compilation thresholds forced low, deoptimize at every eligible point, trigger GC during compilation and installation, invalidate code concurrently, compare optimized results with a reference interpreter, and cap the code cache. Performance tests must include warm-up curves and steady state; reporting only the fastest post-warm-up iteration hides the compilation investment that real users pay.