8.3 Train, Validation, and Test Data

A fitted model can always describe the examples it has already seen more closely as capacity grows. Machine learning instead cares about generalization: performance on future cases drawn from the deployment process. Data splitting creates a rehearsal of that future, but only if the split respects how observations are related.

Give train, validation, and test different jobs

The three partitions are not interchangeable:

A random holdout is a reasonable starting point only when rows are approximately independent and the deployment population resembles the sampled population:

from sklearn.model_selection import train_test_split

X_development, X_final_test, y_development, y_final_test = train_test_split(
    X,
    y,
    test_size=0.20,
    stratify=y,
    random_state=42,
)

stratify=y approximately preserves class proportions. That is useful when the positive class is rare, but it does not solve dependence between rows. Choose a split by asking what the deployed model must generalize to:

• Random split: suitable for approximately independent, identically distributed future rows.

• Stratified split: a random split that preserves target proportions; it is not a substitute for groups or time.

• Group split: keeps every entity—customer, patient, device, store, or document—on one side. Use it when deployment includes unseen entities or multiple rows per entity share information.

• Temporal split: trains on the past and evaluates on later observations. Use it when the model will predict the future and data definitions, behavior, or prevalence can drift.

For unseen customers, group membership must be passed as splitting metadata, not as a target:

from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(
    n_splits=1,
    test_size=0.20,
    random_state=42,
)

train_positions, test_positions = next(
    splitter.split(X, y, groups=orders["customer_id"])
)

train_customers = set(orders.iloc[train_positions]["customer_id"])
test_customers = set(orders.iloc[test_positions]["customer_id"])
assert train_customers.isdisjoint(test_customers)

For time, sorting is necessary but not always sufficient. Ensure the maximum training timestamp precedes the minimum test timestamp, and consider a gap if features or labels use trailing windows that could overlap the boundary. If one order appears in several derived rows, keep the whole information unit together.

Always preserve split indices, seed, target rates, time ranges, and the reason for the strategy. random_state makes a pseudorandom split reproducible; it does not prove that the particular split is representative. Later, cross-validation will reveal sensitivity to several validation folds.

Treat the final test set as a sealed evaluation

Model development is adaptive. An analyst views a validation result, changes features or hyperparameters, and tries again. This deliberately fits choices to validation feedback, so validation performance can become optimistic after many attempts. The final test set provides an independent check only if it does not participate in that loop.

A safe sequence is:

1. Separate the final test rows before exploration that could guide modeling choices.

2. Use only the development portion for feature decisions, preprocessing design, model comparison, hyperparameter search, and threshold selection.

3. Before viewing test results, freeze the prediction contract, feature list, complete pipeline, primary metric, and decision rule.

4. Fit the selected workflow on the permitted development data and evaluate the final test once.

5. Report the prior validation distribution and the final test result together, including important deployment slices.

Model parameters and hyperparameters cross different boundaries. Calling fit learns parameters from a training fold. A search procedure selects hyperparameters from validation-fold results. A final refit learns parameters on all development rows using the selected hyperparameters. None of those steps needs final-test labels.

from sklearn.model_selection import GridSearchCV

search = GridSearchCV(
    estimator=pipeline,
    param_grid={"classifier__C": [0.1, 1.0, 10.0]},
    scoring="average_precision",
    cv=5,
    refit=True,
)
search.fit(X_development, y_development)

# Only after the workflow and reporting plan are frozen:
final_probabilities = search.best_estimator_.predict_proba(X_final_test)[:, 1]

The name X_final_test is useful friction: it reminds reviewers that the object has a special role. Keeping it in a separate file, access-controlled table, or evaluation function can create a stronger barrier for a team.

Looking at final-test performance and making no change is evaluation. Looking, changing the model, and looking again is selection. After enough test-guided changes, the score describes compatibility with this test sample rather than performance on untouched future data. Recovering an unbiased final estimate then requires genuinely new data; renaming the same test set does not reset its history.

One holdout can still be unusually easy or difficult. The next section uses pipelines and cross-validation to repeat the development comparison without moving preprocessing or entity information across fold boundaries.