8.4 Pipelines, Cross-Validation, and Leakage

The workflow now contains several stateful operations: imputation learns fill values, scaling learns location and spread, encoding learns categories, and the estimator learns model parameters. A trustworthy evaluation must redraw the training boundary around all of them for every split. Scikit-learn's Pipeline makes that boundary executable.

Make the fit boundary an object

A Pipeline is an ordered sequence of transformers followed by an estimator. During fit, each transformer fits on its input and transforms it for the next step; the final estimator then fits. During predict, the fitted transformers only transform new $X$, and the fitted estimator predicts—no step learns from those new rows.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ("preprocess", preprocess),
    ("classifier", LogisticRegression(max_iter=1000)),
])

pipeline.fit(X_train, y_train)
valid_probabilities = pipeline.predict_proba(X_valid)[:, 1]

The step names form a parameter path. For example, classifier__C means parameter C on the classifier step, while a nested preprocessing parameter may have several double-underscore levels. This lets search procedures configure a complete workflow without fitting preprocessing separately.

The critical cross-validation sequence is:

for each fold:
    clone the unfitted pipeline
    fit imputation, scaling, encoding, and model on that fold's training rows
    transform and predict that fold's validation rows
    score predictions against validation targets

This is very different from preprocessing all rows once before cross-validation. Suppose the validation fold changes the global mean. A scaler fitted globally encodes that validation information into every transformed training row, even though the estimator itself never receives validation labels. The leak may look small, but the evaluation contract is already broken—and feature selection or target encoding can create far larger optimism.

Pass raw, schema-approved $X$ to the evaluation function:

from sklearn.model_selection import StratifiedKFold, cross_validate

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
splits = list(cv.split(X_development, y_development))

scores = cross_validate(
    pipeline,
    X_development,
    y_development,
    cv=splits,
    scoring={
        "balanced_accuracy": "balanced_accuracy",
        "average_precision": "average_precision",
    },
    return_train_score=True,
)

cross_validate returns arrays, one result per fold. Keep them. A mean without its standard deviation, range, or worst fold can hide an unstable model. Training-fold results also help diagnose a persistent train–validation gap, although they are not a replacement for deployment-slice analysis.

A fitted pipeline is also the deployment artifact: give its predict or predict_proba method raw rows with the declared schema. Reimplementing preprocessing in an API creates training-serving skew, where production transforms inputs differently from evaluation.

Choose folds that simulate deployment, then hunt leakage

In $k$-fold cross-validation, data is divided into $k$ parts. Each part serves as validation once while the other $k-1$ parts train the workflow. Every row receives an out-of-fold evaluation, and the fold distribution shows sensitivity to sampling variation. Five or ten folds are common conventions, not universal laws: more folds cost more computation and do not fix a wrong independence assumption.

The splitter must encode the deployment contract:

For grouped classification, precompute and audit the folds:

from sklearn.model_selection import StratifiedGroupKFold

cv = StratifiedGroupKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)
splits = list(cv.split(X, y, groups=customer_id))

for train_positions, valid_positions in splits:
    train_groups = set(customer_id.iloc[train_positions])
    valid_groups = set(customer_id.iloc[valid_positions])
    assert train_groups.isdisjoint(valid_groups)

Cross-validation reduces dependence on one holdout; it does not automatically prevent leakage. Audit at least four boundaries:

1. Semantic boundary: are all features genuinely available at prediction time, with no target proxies?

2. Fit boundary: does every data-dependent operation—imputation, scaling, encoding, feature selection, resampling—fit inside each training fold?

3. Unit boundary: can related rows, duplicates, customers, or windows cross folds?

4. Selection boundary: were the same cross-validation results used repeatedly to choose among many analyst ideas, and is the final test still untouched?

Use negative controls when a score looks implausibly good. Remove the most suspicious feature and rerun on identical folds. Shuffle $y$ and repeat the full pipeline: performance should fall near the chance baseline. Inspect duplicate hashes and group intersections. Compare performance across time, region, and data-source slices. A negative control does not prove that the workflow is clean, but failure gives a concrete reason to investigate.

Do not select a model from the fold with the highest score. Cross-validation evaluates each candidate over the same series of folds; selection uses the aggregate and its stability. If the project repeatedly tunes both features and hyperparameters against the same cross-validation loop, nested cross-validation can estimate that entire selection procedure with an outer set of folds. It is more expensive and still does not replace a final deployment-representative test.

Chapter checkpoint

A defensible machine-learning workflow is now a chain of explicit contracts:

decision and prediction time
        ↓
target, unit, and horizon
        ↓
time-valid feature allowlist
        ↓
deployment-mirroring train / validation / test split
        ↓
train-only preprocessing + estimator in one Pipeline
        ↓
fold distributions, leakage audits, and one final test

The next machine-learning chapters can now focus on regression, classification, metrics, and model interpretation without confusing a high score with trustworthy evidence.