9.4 Regularization, Diagnostics, and Interpretation

OLS chooses the coefficients that best fit squared error without charging for coefficient size. With many or correlated features, several coefficient combinations can make similar predictions, and small changes in training rows can make the fitted weights jump. Regularization changes the objective to prefer less extreme solutions—but its strength must be selected with validation evidence.

Watch Ridge and Lasso coefficients travel along alpha

Ridge regression adds an $L_2$ penalty:

As $\alpha$ increases, coefficients are continuously pulled toward zero. Ridge often shares predictive weight among correlated features, improving numerical conditioning and reducing coefficient variance at the cost of some bias.

Lasso regression adds an $L_1$ penalty:

Its coefficient paths can reach exactly zero, producing a sparse model. Sparsity is not proof that the surviving variables are the true causes. Among strongly correlated predictors, Lasso may select one representative and change that choice across folds.

alpha is a hyperparameter. Larger values mean stronger shrinkage in both Ridge and Lasso; $\alpha=0$ recovers the unpenalized objective, though LinearRegression is the appropriate estimator for exact OLS. Features should be scaled inside the Pipeline before either penalty is applied.

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV, KFold
from sklearn.pipeline import Pipeline

ridge_pipeline = Pipeline([
    ("preprocess", preprocess),
    ("regressor", Ridge()),
])

cv = KFold(n_splits=5, shuffle=True, random_state=42)
search = GridSearchCV(
    ridge_pipeline,
    param_grid={
        "regressor__alpha": np.logspace(-3, 3, 25),
    },
    scoring="neg_mean_absolute_error",
    cv=cv,
    refit=True,
)
search.fit(X_development, y_development)

Plot two linked curves on a logarithmic alpha axis:

1. every standardized coefficient as a function of alpha;

2. validation error and its fold variation as a function of alpha.

At very weak regularization, variance can remain high. At very strong regularization, coefficients collapse and bias grows. The most useful region is a stable validation-error valley, not necessarily the single lowest pixel on a noisy curve. The final test set does not choose alpha.

Elastic Net combines $L_1$ and $L_2$ penalties. It is a reasonable next candidate when sparsity is useful but correlated features make pure Lasso unstable. It adds another hyperparameter, so it also expands the selection procedure.

Diagnose predictions graphically before interpreting coefficients

One metric and one coefficient table cannot certify a regression model. Build a compact diagnostic report from frozen-validation or out-of-fold predictions:

• actual versus predicted: points should follow the ideal diagonal; range compression, bias, and extreme failures become visible;

• residual versus predicted: inspect curvature, funnels, group bands, and a residual center away from zero;

• residual distribution: inspect skew, heavy tails, and rare large errors while keeping target units visible;

• coefficient distributions across folds: inspect sign, magnitude, and instability rather than trusting one fit.

Scikit-learn can create the first two views from held-out predictions:

import matplotlib.pyplot as plt
from sklearn.metrics import PredictionErrorDisplay

fig, axs = plt.subplots(
    1, 2,
    figsize=(11, 4.5),
    layout="constrained",
)

PredictionErrorDisplay.from_predictions(
    y_true=y_valid,
    y_pred=y_pred,
    kind="actual_vs_predicted",
    ax=axs[0],
)
PredictionErrorDisplay.from_predictions(
    y_true=y_valid,
    y_pred=y_pred,
    kind="residual_vs_predicted",
    ax=axs[1],
)

Interpret coefficients only after predictive quality and stability are acceptable. A multiple-regression coefficient is a conditional predictive association within the chosen features, transformations, population, and regularization. It does not automatically answer what would happen if an operator intervened to change that feature.

Four limitations must remain beside any coefficient chart:

1. Scale: raw magnitudes depend on feature units; standardized magnitudes depend on the training distribution.

2. Correlation: related predictors can exchange weight while predictions remain stable.

3. Omitted variables: leaving out a common cause can change signs or magnitudes.

4. Extrapolation: a linear equation extends indefinitely, but evidence exists only over the observed, deployment-relevant range.

If the target is transformed, such as modeling log1p(y), predictions must be mapped back before reporting minute- or currency-scale errors. TransformedTargetRegressor can keep the forward and inverse target transformations attached to the regressor. Remember that minimizing error in log space answers a different weighting question from minimizing squared error in the original space.

Chapter checkpoint

Regression analysis is now a visual evidence loop:

prediction contract and split
        ↓
encoded, scaled Pipeline
        ↓
OLS / Ridge / Lasso fitted inside training folds
        ↓
MAE, RMSE, R² and slice distributions
        ↓
actual–predicted, residual, and coefficient-path graphics
        ↓
one sealed final evaluation with interpretation limits

Chapter 10 will keep this graph-first workflow while changing the target from a continuous value to a class, where decision boundaries and asymmetric classification errors become central.