9.3 MAE, RMSE, and R-Squared
A regression model produces continuous predictions, so there is no single “correct or incorrect” count. Evaluation must summarize a distribution of residuals. The right summary depends on how the real decision values ordinary misses, severe misses, direction, and performance relative to a baseline.
Let the cost of residuals choose MAE or RMSE
For held-out examples with residuals , mean absolute error (MAE) is
MAE has the target's unit. An MAE of 4.2 minutes says the absolute prediction miss averages 4.2 minutes over this evaluation set. It does not say every order is wrong by 4.2 minutes, and it discards error direction.
Root mean squared error (RMSE) is
The square gives large residuals disproportionate influence; the square root returns the result to the target unit. RMSE is appropriate when severe misses have superlinear operational cost or when it matches the fitted squared-error objective. It is not automatically more “accurate” than MAE.
Consider residuals . The last order contributes 18 units to the absolute-error sum but 324 units to the squared-error sum. A graph of per-row contributions makes the different priorities visible.
from sklearn.metrics import (
mean_absolute_error,
root_mean_squared_error,
)
mae_minutes = mean_absolute_error(y_valid, y_pred)
rmse_minutes = root_mean_squared_error(y_valid, y_pred)Always compute competing models on the identical held-out rows, with the same target definition and sample weights. Report the residual distribution or useful quantiles alongside one average. Two models can share MAE while one has many modest errors and the other has a dangerous failure tail.
Interpret through its mean baseline, then inspect slices
The coefficient of determination, , compares model squared error with a reference that always predicts the evaluation target mean. Define
and
Then
The scale has useful landmarks:
| Interpretation on this evaluation set | |
|---|---|
| predictions exactly match targets | |
| squared error equals the true-mean baseline | |
| below | predictions are worse than that mean baseline |
A negative does not mean negative predictions. Nor does mean “72% accurate.” It means the model reduced squared error by 72% relative to the evaluation mean baseline under this formula. The value has no unit and can change when the target population's spread changes, even if MAE stays similar.
from sklearn.metrics import r2_score
r2 = r2_score(y_valid, y_pred)An aggregate score is a map average; deployment happens at specific locations. Build a row-level prediction table and inspect at least:
- mean residual, because it retains over- versus underprediction direction;
- MAE and RMSE by declared region, period, or customer group;
- residuals across target or prediction quantiles;
- sample size and uncertainty for every slice.
Define important slices and minimum sample sizes before opening the final test set. Otherwise, repeatedly searching for interesting subgroups becomes another form of test-set selection. Small slices are leads for more data, not automatic reasons to rebuild the model.
In scikit-learn cross-validation, losses often use scorer names such as neg_mean_absolute_error because the selection interface assumes larger scores are better. Negate those returned values before presenting them as errors. Keep the complete fold array, not just its mean.
Metrics answer “how much?” but not “what shape of failure?” Section 9.4 combines validation curves, coefficient paths, and residual graphics to choose regularization and diagnose what the averages hide.