8.2 Features, Targets, and Preprocessing

A prediction contract becomes data through two aligned objects: a feature matrix $X$ and a target vector $y$. Building them is not a mechanical “drop the label column” operation. Each row must represent the contract's unit, every feature must exist at prediction time, and every transformation must learn only from permitted training data.

Build $X$ and $y$ with a feature-availability audit

In tabular supervised learning, $X$ normally has shape $(n_{samples}, n_{features})$ and $y$ has one target per sample. Their row alignment is part of the meaning:

feature_columns = ["amount", "zone", "queue_size"]

X = orders.loc[:, feature_columns].copy()
y = orders.loc[X.index, "late"].copy()

assert X.index.equals(y.index)
assert "late" not in X.columns

Use an explicit feature allowlist. orders.drop(columns="late") silently accepts every other column, including a future field added next month. An allowlist makes a schema change visible during review.

Before approving a column, record its source, definition, earliest availability, data type, missing-value behavior, and production owner. Then ask: could the online system produce exactly this value at prediction time? Correlation with the target does not answer that question.

Three feature traps deserve separate labels:

1. Temporal leakage: actual_delivery_minutes is created after dispatch, so it cannot predict delay at dispatch.

2. Target proxy leakage: refund_after_delivery may nearly reveal a late-delivery label. Even if a stored snapshot makes it look available, its business meaning comes from the outcome.

3. Identifier memorization: order_id is unique and usually carries no reusable relation. customer_id may let a random split memorize repeat customers; whether it is valid depends on the deployment population and split strategy.

A small field catalog turns these judgments into auditable data:

catalog = pd.DataFrame({
    "field": ["amount", "zone", "queue_size",
              "order_id", "actual_minutes", "refunded"],
    "available_at": ["order", "order", "dispatch",
                     "order", "delivery", "after_delivery"],
    "role": ["feature", "feature", "feature",
             "identifier", "target", "target_proxy"],
})

Feature availability is about production time, not dataframe position. A column can sit beside the target in a historical table while being generated hours later. Also check whether a feature is stable between training and serving: a warehouse aggregate calculated overnight is not automatically available to a real-time API.

Give each column a train-only preprocessing path

Most estimators expect a numeric feature matrix, but raw tables mix numeric values, missing observations, categories, booleans, and dates. Preprocessing translates each declared input into a stable representation.

Numeric and categorical columns usually need different policies:

Scaling matters for distance-based models and regularized linear models because feature magnitude affects distance or penalty. It usually does not change tree split ordering. This is a model-dependent choice, not a ritual to perform on every number.

For a nominal category such as vehicle_type, arbitrary integers create a fake order: encoding bicycle as 0, car as 1, and scooter as 2 suggests that scooter is “twice” bicycle. One-hot encoding creates separate indicator columns instead. Production can contain a category absent from training, so configure the encoder deliberately:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric_columns = ["amount", "queue_size"]
categorical_columns = ["zone", "vehicle_type"]

numeric_pipeline = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])

categorical_pipeline = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore")),
])

preprocess = ColumnTransformer(
    transformers=[
        ("numeric", numeric_pipeline, numeric_columns),
        ("categorical", categorical_pipeline, categorical_columns),
    ],
    remainder="drop",
)

remainder="drop" is an intentional schema boundary: undeclared fields do not flow into the model. After fitting, get_feature_names_out() helps connect expanded matrix columns back to their sources. One-hot output is often sparse; converting a large sparse matrix to dense form can exhaust memory.

Every value learned by preprocessing is model state. The median of a numeric column, the standard deviation used for scaling, and the category vocabulary must be estimated from the training portion only. This sequence is wrong:

# Wrong: validation rows influence preprocessing state.
X_ready = preprocess.fit_transform(X)
cross_validate(model, X_ready, y, cv=5)

The fix is structural, not a promise to remember the order: put preprocess and the estimator in one Pipeline. Cross-validation can then fit a fresh copy of the whole pipeline inside each training fold. Section 8.4 will inspect that boundary directly.

We now have a declared target, a time-valid feature set, and reproducible preprocessing branches. The next question is which rows are allowed to teach the workflow and which rows must remain independent evidence.