Why Compare PCA Scores With Original Predictors?
Principal component analysis (PCA) replaces a set of correlated predictors with component scores: new variables formed as linear combinations of the original variables. The scores are ordered by how much predictor variation they explain. That ordering can be useful, but it is not an ordering by predictive importance. A component that accounts for little variation in the predictors may still carry a strong relationship with the response.
The practical choice is not simply “PCA or no PCA.” A model may use original predictors, a selected set of component scores, or a combination. The right choice depends on what the model is for. A compact score-based model may be easier to fit when predictors are strongly correlated, while original predictors may be easier to explain and act on. Combining them requires care: if all component scores and all original predictors enter an ordinary linear model, the combined design contains exact redundancy.
What Changes When the Representation Changes?
PCA constructs orthogonal directions that summarize variation in the predictor data. For a centered predictor matrix \(X\), its component directions are eigenvectors of the predictor covariance matrix. The corresponding eigenvalues give the variances along those directions. If \(V\) contains all \(p\) orthonormal component directions, then \(Z=XV\) and \(X=ZV^\mathsf{T}\). Thus, the complete score vector and the original predictor vector determine each other exactly.
If only the first \(k\) scores are retained, with \(k<p\), the mapping back to the original predictors is generally not exact. The omitted directions are information loss from the predictor representation, even if those directions contribute little to total predictor variance. In predictive work, that loss matters only insofar as it affects prediction or another stated objective; “explains most predictor variance” does not by itself establish “predicts the response well.”
The reason follows directly from the linear predictor. If a model using original predictors has linear predictor \(\eta=\beta_0+x^\mathsf{T}\beta\), then \(x=Vz\), so \(\eta=\beta_0+z^\mathsf{T}V^\mathsf{T}\beta\). Conversely, for a full orthonormal \(V\), any coefficient vector on \(z\) can be expressed as a coefficient vector on \(x\). The two parameterizations therefore describe the same linear predictors. This equivalence concerns an unpenalized linear model with all components; it does not imply that a truncated PCA model is equivalent, or that two models with different regularization penalties behave the same.
For a generalized linear model with a linear predictor of this form, the same change-of-coordinates argument applies when all components are retained: the possible linear predictors are unchanged. For other model classes, including tree-based models, equivalent predictions are not guaranteed by this argument. A rotation changes the directions in which a tree can split, for example. Any empirical comparison should therefore use the evaluation method appropriate to the prediction task and apply it consistently across candidate representations.
Worked Example: Full Scores Preserve Information, One Score Does Not
A hypothetical pricing team centers two predictors \(X_1\) and \(X_2\), then expresses each record in two orthogonal coordinates \(U=(X_1+X_2)/\sqrt{2}\) and \(W=(X_1-X_2)/\sqrt{2}\). Four training records have \((U,W)\) values \((-3,-1),(-3,1),(3,-1),(3,1)\). Their response is \(Y=10+2U+W\). Compare a model using the first score alone with models using both scores and the original predictors.
Step 1: Identify the PCA directions and retained variance. Both score columns have mean zero. The mean squared value of \(U\) is \((9+9+9+9)/4=9\), and that of \(W\) is \((1+1+1+1)/4=1\). Their mean product is \((3-3-3+3)/4=0\), so they are uncorrelated. The component variances are 9 and 1. The first component accounts for \(9/(9+1)=0.9\), or 90% of total predictor variance.
Step 2: Fit using the first score alone. The least-squares slope is \(\operatorname{Cov}(U,Y)/\operatorname{Var}(U)\). Since \(Y=10+2U+W\), the covariance is \(2(9)+0=18\), and the variance is 9. The slope is \(18/9=2\); the intercept is 10. Thus the fitted value is \(\hat{Y}=10+2U\). For each record, the residual is \(W\), so the four residuals are \(-1,1,-1,1\). The sum of squared errors is \(1+1+1+1=4\), the mean squared error is \(4/4=1\), and the root mean squared error (RMSE) is \(\sqrt{1}=1\).
Step 3: Fit using both scores. The response is exactly \(10+2U+W\), so the fitted values equal the four observed responses and the residuals are all zero. The training RMSE is 0.
Step 4: Express the same full model using the original predictors. From the definitions, \(U=(X_1+X_2)/\sqrt{2}\) and \(W=(X_1-X_2)/\sqrt{2}\). Substitution gives \(Y=10+(2/\sqrt{2})(X_1+X_2)+(1/\sqrt{2})(X_1-X_2) =10+(3/\sqrt{2})X_1+(1/\sqrt{2})X_2\). This original-predictor model also fits exactly. Adding both scores to both original predictors would not create a richer set of linear predictions: each score is already a linear combination of the originals.
Check and exam trap. The first score captures 90% of predictor variance, but the omitted 10% accounts for all the remaining response variation in this example. The one-score RMSE of 1 and full-model RMSE of 0 confirm the calculation. The common mistake is to treat 90% of predictor variance explained as 90% of predictive information retained.
Comparing Predictive Performance
A sound comparison uses a common prediction target, evaluation data, and metric. For a numerical response, RMSE is the square root of the mean squared prediction error. It is in the response’s units and gives relatively greater influence to larger errors. A lower validation RMSE indicates better predictions on that validation set, but it does not establish that the model will always perform better on new data. Also consider whether the difference is practically meaningful and whether the model can be explained and used as intended.
When a model uses both representations, describe precisely which variables enter it. “Scores plus originals” might mean all original predictors plus all component scores, which is redundant in a linear model, or selected scores plus a selected subset of original predictors, which need not be redundant. In a flexible or penalized model, overlapping information can still affect fitting, feature selection, computation, and interpretation even when it does not add new raw information.
Worked Example: Compare Validation RMSE and Check What “Both” Means
In a hypothetical claim-severity study, three candidate models are evaluated on the same four validation records. One uses original predictors, one uses a retained set of PCA scores, and one uses those scores plus a selected original predictor not fully represented by the retained scores. Actual severities and model predictions, in thousands of dollars, are shown below.
| Record | Actual | Original predictors | PCA scores | Scores plus selected original predictor |
|---|---|---|---|---|
| 1 | 10 | 9 | 11 | 9.5 |
| 2 | 12 | 13 | 11 | 12.5 |
| 3 | 14 | 13 | 15 | 13.5 |
| 4 | 16 | 17 | 15 | 16.5 |
Step 1: Calculate the original-predictor model’s errors. Prediction minus actual gives \(-1,1,-1,1\). The squared errors are \(1,1,1,1\), so the mean squared error is \(4/4=1\), and RMSE is \(\sqrt{1}=1\) thousand dollars.
Step 2: Calculate the PCA-score model’s errors. Its errors are \(1,-1,1,-1\), again giving squared errors \(1,1,1,1\). Its RMSE is also \(\sqrt{4/4}=1\) thousand dollars.
Step 3: Calculate the combined model’s errors. Its errors are \(-0.5,0.5,-0.5,0.5\). The squared errors are \(0.25\) each, for a total of \(1\). Its mean squared error is \(1/4=0.25\), and RMSE is \(\sqrt{0.25}=0.5\) thousand dollars.
Step 4: Interpret the comparison. On these validation records, the combined model has the lowest RMSE: 0.5 thousand dollars, compared with 1 thousand dollars for each alternative. This is a descriptive comparison on the stated validation set, not proof of general superiority. The selected original predictor is not already contained in the retained score set, so it can contribute a direction omitted by the PCA representation.
Check and exam trap. The combined model’s total squared error is 1, not 0.25; 0.25 is the mean squared error. Taking the square root of the mean gives 0.5. A common mistake is to call any model containing both scores and originals a useful “combined” model without checking whether one set is exactly determined by the other.
Low-Variance Components Can Matter
PCA is unsupervised: it orders directions using the predictor values without using the response. This makes it useful for summarizing predictor structure, but it also explains why a variance-based cutoff can remove a predictive signal. A response may depend strongly on a direction along which the predictors vary only slightly. The next example isolates this issue. Its calculations use the same four score-coordinate combinations as before, but the response is related to the lower-variance direction.
Worked Example: The Smallest Component Carries the Signal
Use four centered records with \((U,W)=(-3,-1),(-3,1),(3,-1),(3,1)\), so the component variances are 9 and 1. Suppose the response is \(Y=20+4W\). Compare a model using the first component alone with models using both scores or the original predictors.
Step 1: Fit using \(U\) alone. The covariance between \(U\) and \(Y\) is \(\operatorname{Cov}(U,20+4W)=4\operatorname{Cov}(U,W)=0\). Therefore the fitted slope on \(U\) is \(0/9=0\), and the intercept is the response mean, 20. The predictions are 20 for all four records.
Step 2: Calculate the one-score errors. When \(W=-1\), the actual response is \(20+4(-1)=16\); when \(W=1\), it is \(20+4(1)=24\). In record order, actual responses are \(16,24,16,24\). The residuals, prediction minus actual, are \(4,-4,4,-4\). The squared errors sum to \(16+16+16+16=64\). The mean squared error is \(64/4=16\), so RMSE is \(\sqrt{16}=4\).
Step 3: Fit using both scores. The model \(Y=20+0U+4W\) reproduces every response, giving RMSE 0. Since both original predictors determine both scores, a full original-predictor linear model also reproduces the response exactly.
Check and exam trap. The lower-variance component \(W\) has variance 1, compared with 9 for \(U\), yet it contains the entire response signal. The RMSEs, 4 for \(U\) alone and 0 for the full representation, verify the conclusion. Selecting components solely by cumulative variance explained can therefore harm prediction.
Choose the Representation for Its Intended Use
The models are not interchangeable in every practical sense, even when they span the same linear predictors. A coefficient on an original predictor can often be explained directly in the predictor’s units, subject to the model assumptions and other predictors held fixed. A coefficient on a component score describes a direction made from several predictors; interpreting it requires the component loadings and the scaling used to create the scores. The choice of representation should match the audience and the decision being supported.
| Representation | Potential advantage | Question to check |
|---|---|---|
| Original predictors | Effects and operational actions may be easier to connect to named variables. | Are correlated predictors making estimates or explanations difficult? |
| Selected PCA scores | Fewer, orthogonal inputs can provide a compact representation. | Could an omitted, low-variance direction carry response information? |
| Scores and selected originals | Can retain a useful original variable alongside a compressed summary. | Does each added variable contribute information not already represented? |
| All scores and all originals in linear regression | No new linear prediction space beyond either full representation alone. | Is exact linear dependence making coefficients non-unique? |
The purpose also affects how results should be communicated. If the aim is prediction, compare out-of-sample performance and describe the validation setting. If the aim includes explanation, assess whether component-based effects answer the practical question, or whether original variables are needed for a clear account. If both objectives matter, report the trade-off rather than implying that the representation with the lowest observed error is automatically the best overall choice.
A useful discipline is to state the candidate predictor sets explicitly, identify whether the PCA representation is complete or truncated, and check whether any combined inputs are exact linear combinations. Then evaluate the candidates on the same basis and interpret the results in the context of the model’s intended use. This keeps “variance explained,” “predictive performance,” and “ease of use” as distinct criteria rather than treating them as synonyms.
Check Your Understanding
For each problem, distinguish representation, prediction, and interpretation before choosing a conclusion.
- A two-predictor linear model is transformed to use both orthogonal PCA scores, with neither score omitted. Assuming the transformation is full rank and the same unpenalized least-squares method is used, does the model’s possible set of fitted values change? (Answer: No; the full scores are an invertible change of coordinates, so the possible fitted values are unchanged.)
- A retained component set explains 95% of predictor variance. What can be concluded from that fact alone about the model’s predictive performance? (Answer: Nothing definitive; the omitted 5% may contain little, much, or all of the response-related signal.)
- On three validation records, a model’s prediction errors are \(2,-1,1\). Calculate its RMSE in the response units. (Answer: \(\sqrt{(4+1+1)/3}=\sqrt{2}\), approximately 1.414 response units.)
- A linear regression includes all original predictors and all of their full PCA scores. What fitting issue should be anticipated, and what remains possible? (Answer: Exact linear dependence makes the coefficient vector non-unique; fitted values can still be obtained, but the combined model does not add a new linear prediction direction.)
- A two-score PCA model has validation RMSE 3.2, while a model using the two original predictors plus a selected original variable not determined by those scores has RMSE 2.8 on the same validation records. State the comparison and one limitation. (Answer: The expanded model has lower RMSE by 0.4 response units on those records; this comparison alone does not establish that it will generalize better to new data.)