Decide When Principal Components Help Supervised Prediction

Why Compare Principal Components for Prediction?

When predictors are numerous or strongly correlated, a regression fitted to all of them can be difficult to estimate reliably. Principal component analysis (PCA) replaces the original predictors with uncorrelated linear combinations called principal-component scores. A predictive model can then be fitted to some or all of those scores. The practical question is not whether the scores summarize predictor variation well; it is whether a model using them predicts the outcome better on data not used to fit it.

PCA is an unsupervised transformation: it uses the predictors but not the response to choose directions of variation. Consequently, retaining a large share of predictor variation does not by itself establish good predictive performance. This tutorial focuses on comparing models by validation evidence, aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: Let \(X\) be the \(n \times p\) training predictor matrix. After centering its columns, and optionally scaling them, let \(Z\) be the resulting matrix and let \(S\) be its sample covariance matrix. PCA finds eigenvectors \(u_1,\ldots,u_p\) of \(S\), ordered so their corresponding eigenvalues satisfy \(\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0\). The score for component \(j\) is the projection of a row of \(Z\) onto \(u_j\). A model using the first \(k\) scores is a principal-component predictive model.

What the Transformation Does—and Does Not Do

The component directions depend on how the predictors are measured. If predictors are on very different scales, covariance-based PCA can be dominated by variables with large numerical units. Standardizing each predictor using training-set means and standard deviations instead makes PCA equivalent to analyzing the training correlation matrix. The choice should reflect the prediction problem and must be made without using validation or test observations.

Keeping all \(p\) components is a rotation of the centered predictor space, not a reduction in dimension. In ordinary least-squares regression with an intercept, that full rotation gives the same fitted values as regression on the original predictors, provided the predictors have full rank. Dimension reduction occurs only when fewer than \(p\) components are retained. It is that truncation, together with the resulting change in the fitted model, that must be assessed predictively.

For a regression task, one possible validation measure is root mean squared error (RMSE). For validation observations \(i=1,\ldots,m\), with outcomes \(y_i\) and predictions \(\hat y_i\), define:

\operatorname{RMSE} = \sqrt{\frac{1}{m}\sum_{i=1}^{m}(y_i-\hat y_i)^2}.

The competing models must be evaluated on the same validation observations using the same outcome scale and metric. A lower validation RMSE indicates better performance on that validation set under squared-error loss; it does not prove that the model will always perform better on new data.

Key distinction: PCA orders components by the amount of predictor variation they explain. Supervised prediction is judged by how well the model predicts the response. The two orderings need not agree, so component retention should be assessed as part of the predictive procedure.

Compare Models on a Common Validation Set

Worked Example: Comparing Two Predictions for the Same Claims

A hypothetical insurer predicts a claim-severity index for six validation policies. One model uses the original predictors; another uses a reduced set of principal-component scores. The observed indices and predictions are shown below. Which model has lower validation RMSE?

Policy	Observed \(y_i\)	Original-predictor prediction	Component-score prediction
1	10	10	9
2	12	11	13
3	9	10	7
4	14	13	14
5	11	11	12
6	13	13	12

Method. Calculate each model’s validation residuals as observed minus predicted, square them, sum them, and divide by six before taking the square root.

For the original-predictor model, the residuals are \(0,1,-1,1,0,0\). Their squared values are \(0,1,1,1,0,0\), so the sum of squared errors is \(3\). Therefore,

\operatorname{RMSE}_{\text{original}} = \sqrt{\frac{3}{6}} = \sqrt{0.5} \approx 0.7071.

For the component-score model, the residuals are \(1,-1,2,0,-1,1\). Their squared values are \(1,1,4,0,1,1\), which sum to \(8\). Thus,

\operatorname{RMSE}_{\text{components}} = \sqrt{\frac{8}{6}} = \sqrt{\frac{4}{3}} \approx 1.1547.

Conclusion and check. The original-predictor model has the lower RMSE on these six validation cases. The arithmetic checks: \(0+1+1+1+0+0=3\), while \(1+1+4+0+1+1=8\). This comparison does not show that original predictors will always be better; it shows which of these two fitted procedures performed better on this validation set. The common mistake to avoid is reporting a sum of squared errors inconsistent with the listed squared residuals.

Why a Full Component Rotation Preserves Least-Squares Predictions

A short derivation clarifies why PCA itself does not automatically improve a regression fit. Assume a centered predictor matrix has full column rank, and consider least-squares regression with an intercept. If the predictors are transformed by a square orthogonal matrix \(U\), then \(U'U=I\), and the transformed predictors are \(ZU\). For any coefficient vector \(b\), the fitted predictor contribution is \(ZU b\). Set \(a=Ub\). Since \(U\) is invertible, \(a\) can be any coefficient vector, and \(ZU b=Za\). Thus both models can represent exactly the same fitted values. In particular, using all principal components merely changes coordinates.

PCA also produces uncorrelated component scores in the training data. If \(U\) contains the eigenvectors of \(S\), then the score covariance matrix is:

U'SU = \begin{pmatrix} \lambda_1 & 0 & \cdots & 0\\ 0 & \lambda_2 & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & \lambda_p \end{pmatrix}.

Therefore, the variance of score \(j\) is \(\lambda_j\). This explains how the components are ordered. It does not imply that a component with smaller variance is unhelpful for predicting the response. The predictive consequence of dropping components must be checked using data kept out of the fitting process.

Worked Example: Checking the Full-Rotation Equivalence

Suppose a hypothetical severity model uses two centered, scaled predictors \(z_1,z_2\), with fitted regression equation \(\hat y=10+3z_1+z_2\). Define two orthogonal component scores by \(t_1=(z_1+z_2)/\sqrt{2}\) and \(t_2=(z_1-z_2)/\sqrt{2}\). Express the regression in these scores and verify predictions for four validation policies.

Method. Solve for the original predictors: \(z_1=(t_1+t_2)/\sqrt{2}\) and \(z_2=(t_1-t_2)/\sqrt{2}\). Substitute these expressions into the original fitted equation:

\begin{aligned} \hat y &=10+3\frac{t_1+t_2}{\sqrt{2}}+\frac{t_1-t_2}{\sqrt{2}}\\ &=10+2\sqrt{2}\,t_1+\sqrt{2}\,t_2. \end{aligned}

For \((z_1,z_2)=(1,1)\), the original prediction is \(10+3(1)+1=14\). The scores are \((t_1,t_2)=(\sqrt{2},0)\); the component prediction is \(10+2\sqrt{2}\sqrt{2}=14\).

For \((z_1,z_2)=(1,-1)\), the original prediction is \(10+3-1=12\). The scores are \((0,\sqrt{2})\); the component prediction is \(10+\sqrt{2}\sqrt{2}=12\).

For \((z_1,z_2)=(-1,1)\), the original prediction is \(10-3+1=8\). The scores are \((0,-\sqrt{2})\); the component prediction is \(10-\sqrt{2}\sqrt{2}=8\).

For \((z_1,z_2)=(-1,-1)\), the original prediction is \(10-3-1=6\). The scores are \((-\sqrt{2},0)\); the component prediction is \(10-2\sqrt{2}\sqrt{2}=6\).

Conclusion and check. The four predictions agree exactly: \(14,12,8,6\). Keeping both scores has changed the representation, not the predictive model. The trap is to attribute any improvement to “PCA” without stating whether components were dropped or the fitting procedure otherwise changed.

Validation Must Include the PCA Steps

When the number of components is selected using prediction error, PCA and component selection are part of model fitting. For each validation split, estimate the training means and, if used, standard deviations from that split’s training portion. Fit PCA there, transform its validation portion with those same estimates and component directions, fit the predictive model on the training scores, and calculate validation error. Do not recompute PCA using validation predictors: that lets information from the validation cases influence the feature transformation.

The original-predictor model and the PCA model should use the same splits. If several component counts are considered, compare their validation errors using a prespecified procedure; do not choose a component count from test-set performance. A final test set, if available, should be reserved for evaluating the selected procedure rather than repeatedly guiding its choices.

Worked Example: Comparing Models Across Three Validation Folds

A team compares a regression using five original predictors with one using the first two principal components. Three validation folds each contain four observations. Within each fold, scaling and PCA are fitted using that fold’s training observations only. The squared-error sums on the validation folds are \(5,4,6\) for the original-predictor model and \(3,4,4\) for the two-component model. Compare the pooled validation RMSEs.

Method. Pool the squared errors because each fold has the same four-observation size. There are \(3 \times 4=12\) validation predictions in total.

For the original-predictor model, total squared error is \(5+4+6=15\), giving

\operatorname{RMSE}_{\text{original}} = \sqrt{\frac{15}{12}} = \sqrt{1.25} \approx 1.1180.

For the two-component model, total squared error is \(3+4+4=11\), giving

\operatorname{RMSE}_{\text{PCA}} = \sqrt{\frac{11}{12}} \approx 0.9574.

Conclusion and check. The two-component procedure has lower pooled validation RMSE in this comparison. The sums check directly: the fold totals are \(15\) and \(11\), and the denominator is \(12\) for both. This is evidence in favor of the PCA procedure for these validation splits, not a guarantee of future superiority. A common mistake is to fit the PCA transformation once using all observations before forming folds; that allows validation predictors to influence the transformation.

When PCA Is Worth Considering

PCA is a reasonable candidate when predictors are numerous, strongly correlated, or when a lower-dimensional representation is operationally useful. It can replace a collection of correlated input columns with orthogonal scores, which can make a predictive fit less dependent on estimating separate effects for nearly redundant predictors. But the benefit is empirical: compare the complete PCA procedure against a suitable original-predictor procedure using the same validation design.

A small validation difference should be interpreted cautiously, especially when the validation sample is limited. Consider the prediction task, the error metric, and the stability of the comparison across splits. Also confirm that transformations were fitted within each training split and that any choice of component count was made without using the final test data.

All components retained: this is a rotation, and ordinary least-squares predictions are unchanged when the full-rank assumptions hold.
Some components retained: this reduces dimension, but changes which predictor directions the model can use.
Decision: choose based on validation performance and practical needs, not solely on the percentage of predictor variance explained.

Key takeaway: PCA helps supervised prediction only if the fitted procedure using component scores performs better for the predictive goal. Fit preprocessing and PCA within training data, compare on identical validation cases, and distinguish a full rotation from actual dimension reduction.

Check Your Understanding

For each problem, compare the procedures using the stated information and avoid using evaluation data to fit transformations.

A validation set has four outcomes \(4,7,5,8\). Original-predictor predictions are \(5,7,4,8\), and component-model predictions are \(4,6,6,7\). Calculate both RMSEs and identify the lower one. (Answer: Original SSE \(=2\), RMSE \(=\sqrt{0.5}\approx0.7071\); component SSE \(=3\), RMSE \(=\sqrt{0.75}\approx0.8660\); original-predictor model is lower.)
A PCA model retains all four components from four full-rank predictors and uses ordinary least squares with an intercept. Does this alone reduce the predictor dimension, and should its fitted predictions differ from the original-predictor least-squares fit? (Answer: No dimension reduction; the full orthogonal rotation gives the same fitted predictions.)
Two validation folds each contain five cases. An original-predictor model has fold squared-error sums \(6\) and \(4\); a component model has sums \(5\) and \(3\). Calculate the pooled RMSEs. (Answer: Original \(\sqrt{10/10}=1\); component \(\sqrt{8/10}\approx0.8944\); the component model is lower.)
A candidate standardizes all predictors and estimates PCA once using the complete data, then performs cross-validation on the resulting scores. What is the problem, and what should be done instead? (Answer: The validation-fold predictors influenced scaling and PCA; estimate scaling and PCA using each fold’s training portion, then apply that transformation to its validation portion.)
Across six validation observations, the original-predictor model has absolute errors \(0,1,1,0,2,0\); the PCA model has absolute errors \(1,1,1,1,1,1\). Which has lower RMSE? (Answer: Original SSE \(=0+1+1+0+4+0=6\), RMSE \(=1\); PCA SSE \(=6\), RMSE \(=1\); they tie.)

What You'll Learn