10.4 Thresholds, Curves, and Calibration: From Ranking to Action

Section 10.3 showed that one threshold creates one confusion matrix. A different threshold changes every hard metric even when the fitted model and probability ranking remain untouched. We therefore need three separate decisions: choose a model that ranks useful cases, check whether its probabilities are trustworthy, and choose an action threshold aligned with consequences.

All three decisions belong to development data. The final test remains sealed until the model, calibration method, threshold, positive label, primary metric, and slice plan are locked.

Trace every threshold before choosing one

For one fixed set of scores, lower the threshold from 1 toward 0. At first almost nothing is predicted positive. As the threshold crosses each score, another case enters the positive-action set. The confusion matrix changes one case at a time, producing a path of operating points.

A receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate:

The ROC area under the curve (ROC AUC) summarizes how well the score ranks a randomly selected positive above a randomly selected negative. A diagonal ranking has AUC near 0.5; stronger rankings bend toward the upper-left. AUC does not say which threshold to deploy or whether a probability of 0.8 is numerically honest.

A precision–recall (PR) curve plots precision against recall over thresholds. It focuses directly on positive retrieval. Its no-skill reference depends on positive prevalence, so PR curves from populations with different class rates are not directly comparable without that context. Average precision (AP) summarizes the precision obtained as recall increases.

The paired lab uses one orange threshold point in both coordinate systems. Move it and watch the point travel while ROC AUC and AP stay fixed. Then change prevalence without changing the underlying ranking separation: the ROC view changes modestly, while the PR baseline and precision move visibly.

For development data, use out-of-fold probabilities so each row is scored by a model that did not train on its target:

from sklearn.model_selection import StratifiedKFold, cross_val_predict

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probability = cross_val_predict(
    pipeline,
    X_development,
    y_development,
    cv=cv,
    method="predict_proba",
)[:, positive_column]

Then evaluate thresholds against an explicit decision rule. If an FP costs 2 units and an FN costs 12, the empirical development cost is

Plot cost, precision, recall, and action volume against $t$. Select a threshold from those development curves, freeze it, refit the selected Pipeline on all development rows, and evaluate the final test once. Also report the conventional 0.5 action as a transparent reference.

For calibrated probabilities and a simple choice between “act” and “do not act,” a theoretical threshold can be derived. If acting unnecessarily costs $C_{FP}$ and missing a positive costs $C_{FN}$, act when

That formula assumes the probabilities are calibrated and the cost model is complete. Capacity constraints, benefits, intervention effectiveness, delayed feedback, and unequal case values can require a richer policy. It is a starting point, not a substitute for domain review.

Check whether predicted probabilities mean what they say

Discrimination and calibration can disagree. A model may rank every late order ahead of every on-time order but report 0.99 and 0.01 when the true group frequencies are 0.75 and 0.25. Its ranking is excellent; its probabilities are overconfident.

A calibration curve, or reliability diagram, groups similar predicted probabilities and compares each bin's mean forecast with its observed positive frequency. Perfect calibration lies near the diagonal. A point at $(0.7, 0.5)$ means cases forecast near 70% were positive only about half the time in that evaluation sample.

Bins create a bias–variance tradeoff. Too few hide shape; too many contain little data and jump noisily. Show bin counts, state whether bins use equal widths or equal counts, and inspect important slices. Calibration is a population property: a model calibrated before a prevalence shift may no longer be calibrated after deployment changes.

Two proper probability losses complement the diagram. The Brier score is mean squared probability error,

while log loss penalizes confident errors more severely. Both mix calibration with other aspects of predictive performance, so interpret them beside ranking metrics and the reliability plot rather than calling either a pure calibration statistic.

The final lab offers calibrated, overconfident, and underconfident scenarios. Bubble size reveals bin support. Recalibrate on separate development data and watch points move toward the diagonal; then change FP and FN costs and see the derived action threshold move. The two controls solve different problems.

Scikit-learn can fit post-hoc calibration with CalibratedClassifierCV:

from sklearn.calibration import CalibratedClassifierCV

calibrated = CalibratedClassifierCV(
    estimator=base_pipeline,
    method="sigmoid",
    cv=5,
)

Sigmoid calibration fits a smooth parametric mapping and is often a stable starting point. Isotonic calibration is more flexible and can overfit when calibration data are limited. In both cases, the calibrator must be trained on predictions from data not used to fit the corresponding base estimator. Nested or properly cross-fitted development evaluation preserves that boundary.

Calibration cannot repair poor ranking: a monotonic map may improve probability honesty while leaving ROC AUC essentially unchanged. Conversely, moving a threshold does not calibrate a model. Keep the final report separated into four evidence blocks:

1. Ranking evidence: ROC AUC, PR curve, AP, and fold variation.

2. Probability evidence: reliability diagram with bin counts, Brier score, and log loss.

3. Decision evidence: locked threshold, confusion matrix, precision, recall, action volume, and expected cost.

4. Robustness evidence: prevalence, time or group slices, uncertainty, drift triggers, and known limitations.

That separation completes the classification workflow. We began with a score, transformed it into probability, explored linear and tree-based geometry, counted threshold errors, traced every operating point, and finally connected trustworthy probabilities to a declared business action. The sealed test now evaluates a decision that was fully specified before its labels were opened.