10.1 Logistic Regression: From Scores to Decisions

Chapter 9 predicted a continuous delivery time. The safety rails remain unchanged—features must exist at decision time, preprocessing is fitted inside training folds, candidates are selected on development data, and the final test stays sealed—but the target changes. We now predict whether an order will be late within 45 minutes of dispatch, where late=1 is the positive class and on_time=0 is the negative class. This is binary classification.

The prediction contract must say more than “late or not.” We want $P(Y=1\mid X)$ at dispatch time so operations can choose an action threshold from staffing costs. Probability estimation and action are related, but they are not the same task.

Turn a linear score into a probability

A linear model first combines features into a score:

The score ranges from $-\infty$ to $+\infty$, so it is not yet a probability. Logistic regression passes it through the sigmoid function:

When $z=0$, $p=0.5$. Positive scores produce probabilities above 0.5; negative scores produce probabilities below 0.5. The curve becomes flat near 0 and 1, so another unit of score changes probability most strongly near the middle.

The inverse view is equally useful. The odds are $p/(1-p)$, and the log-odds, or logit, are linear:

Holding other features fixed, increasing $x_j$ by one unit multiplies the odds by $e^{w_j}$. This is a conditional predictive association, not a causal effect. Scaling changes the unit represented by a coefficient, and correlated features can redistribute weight just as they did in regression.

Use the first lab as a live coordinate transform. Move $x$, $w$, and $b$ and follow one point from a linear score to the sigmoid curve. Then move the orange threshold: the probability stays fixed while the action changes.

The model learns coefficients by minimizing log loss, also called binary cross-entropy. For one row,

A confident correct probability receives little loss. A confident wrong probability receives a very large loss, which is why probability quality matters even when two models make the same hard predictions at 0.5. Compare against DummyClassifier(strategy="prior"): it predicts the training-fold class rate and tests whether features add useful information.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

classifier = Pipeline([
    ("preprocess", preprocess),
    ("model", LogisticRegression(max_iter=1000)),
])

classifier.fit(X_train, y_train)
classes = classifier.named_steps["model"].classes_
positive_column = list(classes).index(1)
validation_probability = classifier.predict_proba(X_valid)[:, positive_column]

Locating the column from classes_ makes the positive-label contract explicit. Blindly assuming that the second column always means the desired business event is a fragile habit.

LogisticRegression is regularized by default. Its parameter C is the inverse of regularization strength: smaller C means stronger shrinkage. Scaling numeric inputs inside the Pipeline makes that penalty act on comparable coordinates. Tune C on the same development folds as every other model choice; do not let the final test choose it.

Read a decision boundary without confusing it with probability

A probability becomes a hard prediction only after choosing a threshold $t$:

At the conventional $t=0.5$, the boundary for two features is

That equation draws a straight line. Points on one side are predicted positive; points on the other side are predicted negative. Color bands around it show something the line alone hides: confidence changes continuously across space.

For another threshold, the boundary becomes

Changing $t$ translates the action boundary but does not refit the model or change probability ranking. A dispatch team that considers a missed late order much more costly than an unnecessary alert may choose $t<0.5$. Chapter section 10.4 will tune that decision with explicit costs and out-of-fold probabilities.

The second lab renders the probability surface behind the boundary. Rotate the weights, move the threshold, and then switch the truth to an XOR pattern. No single straight line can capture two diagonally opposite positive regions. That failure is geometric, not a reason to keep rotating the line forever.

Interactions or polynomial features can bend the boundary while retaining logistic regression:

from sklearn.preprocessing import PolynomialFeatures

nonlinear_logistic = Pipeline([
    ("preprocess", preprocess),
    ("basis", PolynomialFeatures(degree=2, include_bias=False)),
    ("model", LogisticRegression(C=0.5, max_iter=1000)),
])

That added capacity must be justified and validated. A beautiful training boundary can simply trace noise. For multiclass targets, logistic regression estimates multiple class scores and predict_proba returns one probability per class; the class order still comes from classes_. We stay binary here so the geometry, positive label, and error costs remain visible.

Logistic regression gives us a smooth global probability surface. The next section studies trees, which replace one global equation with nested if/then regions, and ensembles, which combine many such models to improve stability or correct remaining errors.