10.3 Classification Metrics: Count the Consequences

The previous sections produced probabilities, rankings, and model families. Metrics do not become meaningful until we state a positive class and turn probabilities into actions. We continue to define late=1 as positive. At a chosen threshold, every validation order enters exactly one cell of a confusion matrix.

Build metrics from the confusion matrix

A confusion matrix crosses actual class with predicted class. Scikit-learn uses actual classes as rows and predicted classes as columns. With labels=[0, 1], the binary matrix is

The four cells have operational meanings:

That orientation is worth writing on every plot. A transposed matrix has the same four numbers but answers different row and column questions.

Accuracy measures the fraction of all correct actions:

Precision asks: among predicted positives, what fraction truly was positive?

Recall, also called sensitivity or true-positive rate, asks: among actual positives, what fraction did we catch?

Specificity, or true-negative rate, asks the corresponding negative-class question:

The threshold lab starts above the matrix with every order located on a probability strip. Move the threshold and watch individual points change predicted color before the corresponding cell count changes. This is the causal bridge between a model score and a metric.

The F1 score is the harmonic mean of precision and recall:

F1 becomes high only when both components are high, but it ignores true negatives and assigns FP and FN a particular symmetric tradeoff. That is not automatically the delivery operation's cost function. If a missed late order costs six times an unnecessary alert, report explicit expected cost alongside F1.

from sklearn.metrics import (
    ConfusionMatrixDisplay,
    classification_report,
    confusion_matrix,
)

prediction = (validation_probability >= 0.50).astype(int)
matrix = confusion_matrix(y_valid, prediction, labels=[0, 1])
ConfusionMatrixDisplay(
    matrix,
    display_labels=["on time (0)", "late (1)"],
).plot()

print(classification_report(
    y_valid,
    prediction,
    labels=[0, 1],
    target_names=["on time", "late"],
    zero_division=0,
))

If no rows are predicted positive, precision has a zero denominator. Software needs an explicit reporting policy, but the substantive conclusion is “this threshold triggered no positive actions,” not “precision is safely zero.” Always report support—the number of true examples for each class—beside per-class metrics.

Expose imbalance and slice failures

Suppose only 2% of orders are late. An always-on-time classifier reaches 98% accuracy while catching no late order at all. That is why accuracy must be compared with class prevalence and a DummyClassifier baseline. Balanced accuracy averages recall for the positive and negative classes, so the majority class cannot dominate it through count alone.

Multiclass or multilabel reports introduce averaging choices. Macro averaging computes a metric independently per class and gives each class equal weight. Weighted averaging weights class results by support. Micro averaging pools all cell contributions before computing the metric. None is universally correct: each encodes a different question, and a single aggregate should never replace the per-class table.

Even a good aggregate can hide a bad operational slice. The model might catch late orders in central zones but miss them in rural zones, during storms, or for a less common vehicle. Build a row-level audit table containing the frozen prediction, truth, probability, group, and decision-time context. Then report slice sample size, prevalence, FP, FN, precision, and recall.

The precision–recall lab makes two hiding mechanisms visible. First, lower prevalence depresses precision even when score separation is unchanged because false alarms compete with fewer true positives. Second, raising the threshold often increases precision while decreasing recall. The live predicted-positive count shows the staffing volume behind that tradeoff.

Slice metrics have uncertainty. A region with two positive examples can show 0% or 100% recall from one changed order. Mark sample sizes, predeclare minimum support, and show intervals or fold variation when decisions depend on the slice. Do not delete the majority class simply to make a report look balanced. If training-time resampling is justified, it must occur inside each training fold and evaluation must retain the real deployment distribution.

There are now three different objects in play:

This section measured one threshold. The final section deliberately removes that fixed threshold, traces all possible operating points, checks whether probabilities mean what they claim, and then selects an action without opening the final test.