8.1 Prediction Problems and Model Families
The previous chapters built trustworthy variables and visual explanations. Machine learning adds a new promise: use patterns in known examples to make a useful prediction for an example whose outcome is not known yet. The phrase not known yet is the center of the workflow. A model is only useful if it can run at the real decision time without borrowing information from the future.
Write the prediction contract before choosing an algorithm
A request such as “use AI to reduce late deliveries” is a goal, not yet a modeling problem. Turn it into a prediction contract with five explicit parts:
| Contract field | Delivery example | Why it matters |
|---|---|---|
| Unit | one dispatched order | defines one row and one prediction |
| Prediction time | immediately after dispatch | sets the feature-availability boundary |
| Target | delivered later than 45 minutes | defines the label |
| Horizon | the next 45 minutes | says when the outcome becomes known |
| Action | consider early reassignment | gives the prediction a purpose |
The resulting question is: “For each order at dispatch time, will delivery take more than 45 minutes?” Its target has two discrete outcomes, so it is a binary classification problem. If the target were the number of delivery minutes, it would be regression because the output is continuous. Predicting one of several discrete outcomes—such as low, medium, or high risk—is multiclass classification.
In supervised learning, historical examples contain both inputs and a known target . The model learns a function
where is a prediction, not the observed truth. In unsupervised learning, there is no designated target ; clustering customers into similar groups is an exploration problem, not a late-delivery classifier. Unsupervised results can be useful, but they do not answer a supervised question automatically.
The contract also prevents a subtle mismatch between prediction and action. Predicting at delivery time may be accurate, but it is too late to reassign a driver. A 45-minute risk score may be mathematically valid, but it is operationally empty if nobody can act on it. Ask four questions before opening scikit-learn:
1. What exactly receives one prediction?
2. At what instant must the prediction exist?
3. When and how is the target confirmed?
4. Who changes what decision because of the output?
Compare useful biases, capacity, and a dummy baseline
An algorithm is not a neutral container. Every model family has an inductive bias: assumptions that make some patterns easier to learn than others.
- A linear regression model assumes the prediction can be expressed as a weighted sum of features. Logistic regression applies a linear decision rule to class probabilities. These models are fast, regularizable, and often interpretable, but a purely linear boundary cannot represent every interaction.
- A decision tree repeatedly splits feature space into regions. It naturally expresses thresholds and interactions, but an unrestricted tree can memorize small details.
- A nearest-neighbor model predicts from similar training examples. It can represent local shapes but depends strongly on feature scale and becomes expensive or unreliable in high dimensions.
- Ensembles combine multiple models to reduce variance or bias. They can be powerful, but more machinery does not repair a broken target, leaky feature, or unrealistic split.
Start every experiment with a dummy baseline. DummyRegressor can predict the training mean or median; DummyClassifier can predict the most frequent class or generate predictions from the training class distribution. A learned model that does not reliably beat this simple rule has not yet demonstrated useful signal.
from sklearn.dummy import DummyClassifier
baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
baseline_predictions = baseline.predict(X_valid)Model capacity describes how flexible a model is. Too little capacity creates underfitting: training and validation performance are both weak because the model cannot express the signal. Too much capacity can create overfitting: training performance rises while validation performance stalls or falls because the model learns sample-specific noise.
Compare training and validation results on the same split:
| Pattern | Likely diagnosis | Useful response |
|---|---|---|
| train low, validation low | underfitting or weak features | improve features or add appropriate capacity |
| train high, validation much lower | overfitting / high variance | regularize, reduce capacity, or add representative data |
| both beat baseline with a modest gap | plausible generalization | verify across folds and deployment slices |
For a regularized model, a hyperparameter such as tree depth or regularization strength is chosen by the analyst; a fitted parameter such as a coefficient or split threshold is learned from training data. Do not select either model family or hyperparameters from the final test score. Chapters 8.3 and 8.4 will build the separation that keeps selection honest.
There is no universally best family. The right first comparison uses a small set of contrasting, defensible biases under identical data splits and metrics: a dummy rule, a simple linear model, and a restrained nonlinear model are often more informative than a large leaderboard.
The contract tells us what may be predicted and when. The next section turns that boundary into a feature matrix without allowing the target, the future, or preprocessing statistics to sneak into .