11.2 Dimensionality Reduction with PCA

Section 11.1 showed that redundant, differently scaled coordinates can distort distance. High-dimensional data also resists direct visualization. Principal component analysis (PCA) builds new orthogonal coordinates that preserve as much variance as possible in fewer dimensions.

PCA is unsupervised and linear. It sees feature covariance, not target relevance, causality, or cluster truth. Its value is geometric compression, visualization, noise reduction, and sometimes more stable downstream modeling.

Rotate the axes toward maximum variance

Start with centered rows $x_i-\bar{x}$. For a unit direction $w$, the one-dimensional score is

The first principal component, PC1, chooses $w$ that maximizes the variance of these scores. Geometrically, project every point perpendicularly onto a candidate line and rotate the line until the projected dots spread out most. The second component maximizes remaining variance subject to being orthogonal to PC1. Later components follow the same rule.

Equivalently, the component directions are eigenvectors of the covariance matrix, ordered by decreasing eigenvalue. Scikit-learn computes PCA with singular value decomposition and stores the axes in components_. The score matrix returned by transform has one column per retained component.

The projection lab lets you rotate a candidate axis. Gray segments show what each point loses when projected; orange dots are the one-dimensional scores placed back on the line. The current projected variance is compared with the best possible direction. Change feature 2's scale and the best axis rotates—visible proof that scaling is a modeling choice.

Scikit-learn PCA centers columns but does not automatically standardize their variance. If features use different units and those units should not define importance, place StandardScaler before PCA:

from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

pca_pipeline = make_pipeline(
    StandardScaler(),
    PCA(),
)

scores = pca_pipeline.fit_transform(development[features])
pca = pca_pipeline.named_steps["pca"]

Scaling and PCA must be fitted inside the same valid data boundary. If PCA feeds a supervised model evaluated by cross-validation, both transformations belong inside the Pipeline fitted separately in each training fold. Fitting PCA once on all rows leaks validation distribution, even though PCA never sees $y$.

The component coefficients are often called loadings. A large positive or negative coefficient means that feature strongly aligns with the component direction. Signs are arbitrary: multiplying a whole component and all its scores by $-1$ describes identical geometry. Interpret relative patterns and subspaces, not whether software happened to point an axis left or right.

Decide what compression discards

For component $j$, explained_variance_ratio_[j] is its share of total variance. A scree plot shows individual shares; a cumulative curve shows how much the first $m$ components retain:

Choosing the smallest $m$ above 90% or 95% can be a useful starting rule, not a universal optimum. Variance is not business value. A rare fraud direction, sensor fault, or minority-group pattern may have little total variance and still matter greatly.

PCA can return retained scores to the original feature space with inverse_transform. With all components, reconstruction is essentially exact apart from numerical precision. With fewer components, each row is reconstructed as its projection onto the retained subspace. The difference is reconstruction error.

The reconstruction lab behaves like a two-channel codec. Inspect four component signatures, choose any two channels to transmit, and compare every record before and after decoding. Most rows reward the high-variance components, but two rare rows carry an important contrast in a low-variance direction. Overall RMSE and rare-signal error can therefore disagree, demonstrating why a high cumulative percentage alone cannot authorize compression.

Evaluate compression from several views:

• Plot individual and cumulative explained variance.

• Measure held-out reconstruction error in original business units, per feature and per important group.

• Inspect PC score plots for outliers, gradients, and overlap without inventing clusters from colors.

• Check component or subspace stability across folds or bootstrap samples.

• Evaluate the actual downstream model on valid folds if prediction is the goal.

PCA is linear, so a curved manifold may still need many components. It is also sensitive to extreme points because variance is squared distance from the mean. Robust scaling, record verification, or another dimensionality method may be appropriate, but every alternative brings its own assumptions.

Never claim that PC1 “causes” a pattern or that two visible clouds prove natural classes. A two-dimensional PCA chart is a projection: overlap may be hidden in discarded dimensions, and separation may be exaggerated by axis limits or selective coloring.

The first half of Chapter 11 treated rows as an unordered cloud. The next section adds a special coordinate that cannot be shuffled away: time. We will build trustworthy time indexes, calendar bins, and rolling windows before attempting any forecast.