9.1 Linear Regression: Intuition and Assumptions

Chapter 8 built the safety rails: a prediction contract, time-valid features, deployment-mirroring splits, and one Pipeline fitted inside each training fold. We can now study a model without losing those rails. Our running contract is to predict delivery duration in minutes for each newly dispatched order. Because the target is continuous, this is regression.

See ordinary least squares as a geometric search

Begin with a visual baseline. Put route distance on the horizontal axis and delivery duration on the vertical axis. Each dot is one training order. A one-feature linear model draws a line:

where $\hat{y}_i$ is the predicted duration for order $i$, $\beta_0$ is the intercept, and $\beta_1$ is the slope. If $x$ is measured in kilometers and $y$ in minutes, the slope has units of minutes per kilometer. The intercept is the prediction at $x=0$; it can be mathematically necessary even when that particular input is not operationally meaningful.

The vertical signed distance from a point to its prediction is a residual:

A positive residual means the model underpredicted; a negative residual means it overpredicted. Ordinary least squares (OLS) chooses coefficients that minimize the residual sum of squares:

Squaring prevents positive and negative residuals from cancelling, and makes long residuals especially expensive. In the lab, each residual becomes the side of a square: OLS chooses the line with the smallest total square area.

With several features, the same idea becomes a plane or hyperplane:

LinearRegression estimates these coefficients from training data:

from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline

dummy = Pipeline([
    ("preprocess", preprocess),
    ("regressor", DummyRegressor(strategy="mean")),
])

linear = Pipeline([
    ("preprocess", preprocess),
    ("regressor", LinearRegression()),
])

dummy.fit(X_train, y_train)
linear.fit(X_train, y_train)
validation_predictions = linear.predict(X_valid)

The mean dummy is still necessary. A fitted line is not useful merely because it has coefficients; it must improve held-out predictions over a rule that ignores $X$.

Read assumptions as patterns in held-out residuals

“Linear” means linear in the coefficients, not necessarily a straight relationship in every raw input. PolynomialFeatures can create $x^2$ or interaction terms such as $x_1x_2$; a model remains linear in its learned coefficients:

Do not add terms merely to flatten training residuals. Each transformation belongs inside the Pipeline and must be compared on the same validation folds.

Residual graphics turn modeling assumptions into visible questions:

The useful prediction assumptions are practical rather than ceremonial:

• The conditional mean can be approximated by the chosen feature basis.

• Training rows do not secretly duplicate or reveal validation rows; dependence is handled by the split.

• Features are not perfectly redundant, and strong collinearity is diagnosed because it destabilizes coefficients.

• The training population represents the deployment population well enough for the intended range.

Constant residual variance is important for a uniform error story and for classical uncertainty calculations. A normally distributed target is not required to fit OLS. Some inferential formulas make assumptions about residuals; predictive evaluation should still use held-out evidence.

An outlier is not automatically an error. A point may be a data mistake, a valid rare operation, or evidence that one model is insufficient. Verify its origin before removing it, then show the sensitivity of conclusions with and without any defensible treatment.

OLS has now given us a geometric objective and a diagnostic language. The next section examines how raw categories and incompatible measurement units create the design matrix that this geometry actually sees.