13.4 ABI and Calling Convention Details

An application binary interface (ABI) specifies how independently compiled code interoperates. A calling convention is the function-call portion of that contract: where arguments and results go, which registers survive, how the stack is aligned, and how control returns. The ABI also covers data layout, symbol naming, exception handling, and object-file conventions.

Matching source-language types is not enough. Caller and callee must agree on the exact machine-level classification of every value.

Classifying and placing arguments

Most 64-bit ABIs pass several integer/pointer arguments in general-purpose registers and floating/vector arguments in another register bank, then place overflow arguments on the stack. The exact registers differ:

• System V x86-64 begins integer arguments with rdi, rsi, rdx, rcx, r8, r9;

• Windows x64 begins with rcx, rdx, r8, r9 and reserves 32 bytes of shadow space;

• AArch64 commonly uses x0–x7 for integer arguments and v0–v7 for floating/vector arguments.

Aggregate types require classification. A small structure may be split across registers; a larger return value may use a hidden pointer to caller-allocated storage. A method can pass a hidden receiver, and a closure can pass an environment pointer. These hidden operands consume ABI locations even though they are absent from the source parameter list.

Stack alignment and variadic calls

The stack pointer must satisfy the ABI's alignment rule at each call boundary—commonly 16 bytes on modern 64-bit platforms. Pushing one value or reserving an odd-sized frame can break alignment unless compensated with padding.

Variadic functions need extra protocol because the callee cannot infer all argument classes from a fixed signature. An ABI may require floating arguments to be mirrored, a count of vector-register arguments, or a register-save area that va_start can traverse. Applying ordinary fixed-signature lowering to a variadic call can make arguments invisible to the callee.

Caller-saved and callee-saved registers

Caller-saved registers may be overwritten by a call. A caller with live values in them must save, move, or spill those values before calling. Callee-saved registers must be restored by any callee that uses them, usually through its prologue and epilogue.

This division distributes cost. Short-lived values and leaf functions favor caller-saved registers because no prologue save is needed. Values live across many calls may favor callee-saved registers because the function pays one save/restore pair rather than one around every call.

Tail calls, exceptions, and interoperability

A tail call can reuse the current frame only if argument placement, return convention, stack cleanup, security instrumentation, and unwind rules are compatible. Otherwise the backend must emit an ordinary call and return.

Exception handling adds platform-specific landing pads, personality functions, and unwind tables. A foreign function interface must also handle name mangling, structure layout, endianness, ownership, and which side catches exceptions. Never let exceptions cross an ABI boundary unless both runtimes explicitly support the same mechanism.

ABI tests should compare generated code against small hand-written assembly stubs. Exercise every argument-register boundary, mixed integer/floating arguments, small and large structures, stack alignment, varargs, callbacks, caller/callee saves, exceptions, and tail calls. A one-instruction mistake may appear only when optimized code or another compiler calls the function.