9.2 Feature Encoding and Scaling

The regression equation in Section 9.1 operates on numeric columns. A real delivery table contains kilometers, currency, queue counts, zones, and vehicle types. Encoding and scaling do more than satisfy an API: they define the coordinate system in which the regression line, coefficient penalty, and interpretation exist.

Turn categories into honest geometry

vehicle_type is nominal: bicycle, car, and van have identities but no inherent numeric distance. Encoding them as 0, 1, and 2 forces a line through an invented order. It says car is halfway between bicycle and van, which is not part of the data contract.

One-hot encoding creates one indicator direction per learned category:

The model can now learn a separate offset for each category instead of imposing equal spacing. With an intercept, a complete one-hot set is exactly redundant because the indicator columns sum to one. One conventional parameterization drops one reference level, so remaining category coefficients express differences from that reference. The predictions can remain identifiable even when individual coefficients depend on the chosen parameterization.

Do not confuse dropping one reference indicator with dropping the whole feature. Also avoid interpreting the reference as a universally “normal” group; it is a coordinate choice.

Production may contain a category absent from training. handle_unknown="ignore" maps it to zeros for that encoder branch rather than crashing. This is a defined fallback, not evidence that the model understands the new category. Monitor its frequency and decide how the prediction should behave.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric_columns = ["distance_km", "amount", "queue_size"]
categorical_columns = ["zone", "vehicle_type"]

preprocess = ColumnTransformer([
    ("numeric", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
    ]), numeric_columns),
    ("categorical", Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("onehot", OneHotEncoder(handle_unknown="ignore")),
    ]), categorical_columns),
], remainder="drop")

Category vocabularies and fill values are learned state. They must be fitted inside each training fold, exactly like the regression coefficients.

Scale the coordinate system before penalizing or comparing coefficients

StandardScaler learns each training column's mean $\mu_j$ and standard deviation $s_j$, then transforms a value as

It must learn $\mu_j$ and $s_j$ from the current training fold. A validation value may lie outside the training range; that is allowed. Clipping it merely to make $z$ look familiar would hide possible deployment shift.

Unregularized OLS predictions are generally invariant to a consistent change of units when coefficients and intercept can adjust. Expressing distance in meters instead of kilometers makes its raw coefficient one-thousandth as large, but does not make distance one-thousandth as useful. Raw coefficient magnitude therefore cannot rank features with different units.

Scaling becomes especially important for Ridge and Lasso because their penalties act directly on coefficient magnitudes. Without scaling, the same predictive contribution can require a large coefficient for one feature and a tiny coefficient for another, so regularization shrinks them unequally for an arbitrary reason.

After standardizing numeric inputs, one coefficient describes the predicted target change for a one-training-standard-deviation change in that feature, with other modeled features held constant. That makes numeric coefficient magnitudes more comparable, but not automatically causal or globally stable.

from sklearn.linear_model import Ridge

regression_pipeline = Pipeline([
    ("preprocess", preprocess),
    ("regressor", Ridge(alpha=1.0)),
])

regression_pipeline.fit(X_train, y_train)
feature_names = regression_pipeline[
    "preprocess"
].get_feature_names_out()
coefficients = regression_pipeline["regressor"].coef_

Interpret a multiple-regression coefficient conditionally: it describes the model's change in prediction for one feature while all other modeled features remain fixed. This differs from a marginal scatterplot, where correlated features vary together. If age and experience are correlated, their coefficients can redistribute across folds even while predictions remain similar.

Always visualize coefficient stability across validation folds. A large single fitted coefficient may reflect unit choice, collinearity, sampling noise, or an extrapolated region. Scaling fixes only the unit problem.

The feature matrix now has an explicit geometry. The next section asks how far its predictions miss, how much severe misses should matter, and what “better than baseline” means numerically and visually.