10.2 Trees and Ensembles: Rules, Regions, and Stability

Logistic regression described the whole feature space with one score equation. That global geometry is compact and often strong, but an operations rule may depend on sharp interactions: rain matters only for long routes, and queue size matters differently by vehicle type. A decision tree learns nested if/then rules that divide feature space into local regions.

We keep the same late-order contract, positive label, development folds, and sealed final test. A flexible model does not relax evaluation discipline; it makes that discipline more important.

See a tree as recursive partitioning

At one node, a classification tree considers candidate splits such as distance_km <= 5.2. Each split sends rows left or right. The tree greedily chooses a feature and threshold that most reduce class impurity. With class proportions $p_1,\ldots,p_K$ in a node, common criteria include Gini impurity,

and entropy,

Both are small when a node is dominated by one class. They are split objectives, not deployment metrics. A split can reduce Gini while making the eventual business cost worse, so model selection still uses held-out metrics aligned with the prediction contract.

The recursion creates axis-aligned rectangular regions in two numeric dimensions. All orders arriving at one terminal region, or leaf, share a prediction. For ordinary classification trees, a leaf's probability is the weighted class proportion among training rows in that leaf. A leaf containing 9 late orders out of 10 reports approximately 0.9. That estimate is less trustworthy than the same proportion from 900 out of 1,000.

Tree capacity is controlled by connected choices:

• max_depth limits how many consecutive questions a path can ask.

• min_samples_leaf prevents tiny terminal groups.

• min_samples_split controls whether a node may branch.

• ccp_alpha can prune branches through cost-complexity pruning.

The lab links the rule geometry to train and validation curves. Increase depth and watch orange partitions multiply. Training score continues upward, while validation score eventually flattens or falls. Then enlarge the minimum leaf: some narrow regions disappear and their unstable extreme probabilities are pooled.

from sklearn.tree import DecisionTreeClassifier

tree = Pipeline([
    ("preprocess", tree_preprocess),
    ("model", DecisionTreeClassifier(
        max_depth=5,
        min_samples_leaf=20,
        random_state=42,
    )),
])

Trees do not generally need StandardScaler because split order is unchanged by a monotonic rescaling. They still need leakage-safe imputation and category handling. Scikit-learn's standard decision-tree implementation does not directly accept raw categorical semantics, so nominal values normally require a supported encoding. Keep that transformer inside the Pipeline and learn its vocabulary from each training fold.

A tree diagram can be large and visually persuasive while generalizing poorly. Treat plot_tree as an explanation of a locked candidate, not evidence that the candidate is valid. Also check probability-region maps, validation capacity curves, leaf sample sizes, slice performance, and stability across folds.

Combine trees without hiding how capacity grows

One deep tree is unstable: a modest data change may alter an early split and replace every descendant rule. Ensembles combine many learners. Two important families solve instability differently.

A random forest trains trees independently enough to average them. Each tree sees a bootstrap sample, and each split considers a random feature subset. Those mechanisms make tree errors less correlated. Averaging class probabilities reduces variance when the trees contain useful signal but do not all make the same mistakes.

For $B$ fitted trees,

More trees usually stabilize the average rather than make its functional class dramatically more flexible. Parameters such as min_samples_leaf, max_depth, and max_features still control the constituent trees. Out-of-bag estimates can provide an additional training-time diagnostic for bootstrap forests, but they do not replace a deployment-valid split or the sealed test.

Gradient boosting is sequential. A shallow tree is added, then later trees focus on directions that reduce the current loss. For classification, the stages update a score that is transformed into probability. Learning rate and number of stages trade off with one another: a smaller learning rate often requires more stages.

from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier

forest = RandomForestClassifier(
    n_estimators=300,
    min_samples_leaf=8,
    max_features="sqrt",
    n_jobs=-1,
    random_state=42,
)

boosting = HistGradientBoostingClassifier(
    max_iter=180,
    learning_rate=0.06,
    max_leaf_nodes=15,
    random_state=42,
)

The ensemble lab makes the distinction graphical. In forest mode, thin member curves are fitted in parallel and the heavy curve is their average; resampling changes individual paths more than the average. In boosting mode, each stage modifies what came before. Too many stages can keep adapting after validation evidence has stopped improving.

Compare a tree, forest, boosted model, logistic model, and dummy on identical folds. Report fold distributions rather than just their means. Average precision and ROC AUC inspect ranking; log loss inspects probability quality; threshold metrics inspect an action. A model can win one view and lose another.

Feature importance also needs restraint. Impurity-based tree importance can favor features with many opportunities to split and can distribute credit strangely among correlated variables. Permutation importance on held-out data asks a more direct predictive question—how much score deteriorates when a column is shuffled—but still does not establish causality. Any interpretation must respect feature availability and dependence.

Trees have now converted probability modeling into visible regions, and ensembles have shown how model combination changes stability. The next section fixes an action threshold and counts exactly which predictions are correct or wrong, then turns those counts into classification metrics.