Explained Variance Is Not Predictive Performance

What Explained Variance Does—and Does Not—Measure

Principal component analysis (PCA) orders components by how much variation they capture in the predictor variables. That ordering is useful when the aim is to summarize or reconstruct the predictors. It does not use a response variable, however, so it cannot by itself rank components by their usefulness for predicting an outcome.

This distinction matters when PCA is used to reduce the number of predictors in a model. A component with a small eigenvalue may vary little across records, yet its values may be strongly associated with the outcome. Discarding it because it explains little predictor variance can therefore remove useful predictive information. Conversely, a component can explain substantial predictor variation while having little relationship with the outcome.

Definition: Let $T_1,\ldots,T_p$ be centered principal component scores from a covariance-based PCA, ordered by decreasing variance, with $\operatorname{Var}(T_j)=\lambda_j$. The proportion of total predictor variance explained by component $j$ is $\lambda_j/\sum_{k=1}^p\lambda_k$. This is a property of the predictors and the PCA calculation; it is not a measure of predictive accuracy for an outcome $Y$.

The scores are uncorrelated: for distinct components $j$ and $k$, $\operatorname{Cov}(T_j,T_k)=0$. Uncorrelated does not generally mean independent. Variance calculations can use zero covariances without claiming independence. The examples below make that distinction explicit.

How Component Variance Relates to Linear Predictive Signal

Suppose a centered outcome follows a linear model in the component scores: $Y=\alpha+\sum_{j=1}^p\beta_jT_j+\varepsilon$, where $E[\varepsilon]=0$, $\operatorname{Cov}(T_j,\varepsilon)=0$ for every $j$, and $\operatorname{Var}(\varepsilon)=\sigma^2$. These assumptions say that the residual has zero covariance with each score; they do not require the scores to be independent of one another.

Under these assumptions, the component’s contribution to the variance of the linear signal is $\beta_j^2\lambda_j$. A component’s predictive signal depends on both its variance and its relationship with the outcome, represented here by $\beta_j$. The eigenvalue alone is not enough.

Result: For the stated linear model, with pairwise uncorrelated scores and residual uncorrelated with every score, $$ \operatorname{Var}(Y)=\sum_{j=1}^p\beta_j^2\lambda_j+\sigma^2. $$ If a model retains only components in a set $S$, its population residual is the sum of the omitted terms $\beta_jT_j$, for $j\notin S$, and $\varepsilon$. Its mean squared prediction error is $\sum_{j\notin S}\beta_j^2\lambda_j+\sigma^2$.

These formulas describe a linear prediction setting with the stated covariance conditions. They are not a guarantee that a linear model is appropriate for every outcome, nor do they describe every possible nonlinear relationship. In particular, PCA’s ranking still does not assess whether a component is useful through a threshold, interaction, or other nonlinear pattern.

Derivation: Why the Omitted-Component Error Adds Up

Let $S$ be the set of retained components. With the population coefficients known, the prediction is $\widehat{Y}_S=\alpha+\sum_{j\in S}\beta_jT_j$, so the residual is $Y-\widehat{Y}_S=\sum_{j\notin S}\beta_jT_j+\varepsilon$. Each score has mean zero, as does $\varepsilon$. Expand the variance of this sum: it is the sum of the individual variances plus twice the sum of all pairwise covariances. Every covariance between two distinct score terms is zero because the scores are uncorrelated. Every covariance between a score term and $\varepsilon$ is zero by assumption. Therefore only the individual variances remain:

\operatorname{Var}(Y-\widehat{Y}_S) =\sum_{j\notin S}\beta_j^2\operatorname{Var}(T_j)+\operatorname{Var}(\varepsilon) =\sum_{j\notin S}\beta_j^2\lambda_j+\sigma^2.

This derivation requires uncorrelated terms, not independent terms. In particular, the fact that PCA scores are uncorrelated is sufficient for the score covariances in this calculation to vanish; it does not establish that the scores are independent.

Worked Example: A Low-Ranking Component Carries Outcome Signal

Consider a hypothetical portfolio with three centered PCA scores satisfying $\operatorname{Var}(T_1)=40$, $\operatorname{Var}(T_2)=1$, and $\operatorname{Var}(T_3)=1$. The scores are pairwise uncorrelated. Suppose the centered outcome is $Y=2T_1+3T_2+\varepsilon$, where $\operatorname{Var}(\varepsilon)=9$ and $\varepsilon$ is independent of the score vector. Find the predictor variance explained by the first component, the outcome variance, and the residual variance when using only the first component.

Method. Compare the PCA eigenvalue proportions with each component’s contribution to the outcome variance. Because the scores are uncorrelated and the noise is independent of them, the score terms and noise are pairwise uncorrelated, so their variances add.

The total predictor variance is $40+1+1=42$. The first component explains $40/42=0.9524$, or 95.24%, of that variance. The second and third components each explain $1/42=0.0238$, or 2.38%.

The outcome is the sum of $2T_1$, $3T_2$, and $\varepsilon$. Their variances are $\operatorname{Var}(2T_1)=2^2(40)=160$, $\operatorname{Var}(3T_2)=3^2(1)=9$, and $\operatorname{Var}(\varepsilon)=9$. Thus $\operatorname{Var}(Y)=160+9+9=178$. The component terms are pairwise uncorrelated; no independence between the PCA scores is assumed.

A population linear model retaining only $T_1$ leaves residual $3T_2+\varepsilon$. Its mean squared prediction error is $3^2(1)+9=18$, and its root mean squared error is $\sqrt{18}=4.2426$, rounded to four decimal places. Retaining $T_1$ and $T_2$ leaves residual $\varepsilon$, with mean squared error 9 and root mean squared error 3.

Check. The first component accounts for $160/178=0.8989$, or 89.89%, of the outcome variance in this linear model. Adding the second component raises that proportion to $(160+9)/178=0.9494$, or 94.94%. The second component explains only 2.38% of predictor variance, but its contribution to outcome variance is 9. PCA’s predictor-variance ranking alone does not reveal that contribution.

Common mistake avoided. Do not infer that retaining 95.24% of predictor variance leaves only 4.76% of outcome variance unexplained. That would incorrectly estimate the residual variance as $0.0476(178)=8.4728$. The actual residual variance is 18 because the omitted component’s outcome contribution depends on both its eigenvalue and its coefficient.

Worked Example: Comparing Components by Their Linear Signal

In a hypothetical equipment study, two centered, uncorrelated component scores have variances 9 and 0.25. The outcome model is $Y=0.2T_1+4T_2+\varepsilon$, where the residual is uncorrelated with both scores and has variance 9. Calculate the predictor variance proportions and compare the linear signal contributed by each component.

The total predictor variance is $9+0.25=9.25$. Component 1 explains $9/9.25=0.9730$, or 97.30%, and component 2 explains $0.25/9.25=0.0270$, or 2.70%, rounded to four decimal places for the proportions.

The signal contribution from component 1 is $0.2^2(9)=0.04(9)=0.36$. The signal contribution from component 2 is $4^2(0.25)=16(0.25)=4$. Thus the component explaining only 2.70% of predictor variance contributes more than eleven times as much outcome signal as the first component: $4/0.36=11.1111$.

The full model’s outcome variance is $0.36+4+9=13.36$. If only $T_1$ is retained, the residual contains $4T_2+\varepsilon$, so its mean squared error is $4+9=13$ and its root mean squared error is $\sqrt{13}=3.6056$, rounded to four decimal places. If both components are retained, the error variance is 9 and the root mean squared error is 3.0000.

Check. Adding $T_2$ reduces the population mean squared error from 13 to 9, a reduction of 4, exactly the signal contribution calculated for that component. The large eigenvalue of $T_1$ does not make it the more valuable predictor in this example.

A Low-Variance Component Can Be Useful Beyond a Linear Model

The mismatch is not limited to linear models. A component can carry outcome information through a rule that is not well summarized by its variance or by a linear coefficient. The following deliberately simple hypothetical example isolates that point.

Worked Example: A Small-Variance Score Separates Two Outcome Groups

Suppose four equally likely records have component scores $(T_1,T_2)$ equal to $(5,0.5)$, $(5,-0.5)$, $(-5,0.5)$, and $(-5,-0.5)$. Define a binary outcome $Y$ to be 1 when $T_2=0.5$, and 0 otherwise. Find the component variances, the first component’s explained-variance proportion, and the classification error from using either score alone with the best available rule.

Each score has mean zero. The four equally likely combinations give $\operatorname{Var}(T_1)=25$ and $\operatorname{Var}(T_2)=0.25$. Their covariance is zero: the products $T_1T_2$ are $2.5,-2.5,-2.5,2.5$, whose average is 0. Thus these are uncorrelated scores, with $T_1$ first because its variance is larger.

Total predictor variance is $25.25$. The first component explains $25/25.25=0.9901$, or 99.01%; the second explains $0.25/25.25=0.0099$, or 0.99%, both rounded to four decimal places for the proportions.

A rule that predicts $Y=1$ when $T_2>0$ and $Y=0$ otherwise classifies all four records correctly, so its classification error is $0/4=0$. By contrast, each value of $T_1$ occurs once with $Y=1$ and once with $Y=0$. Given either value of $T_1$, the two outcomes are equally likely. A classifier using only $T_1$ cannot distinguish the outcomes; either constant prediction makes two errors out of four, for an error rate of $2/4=0.5$.

Check. The second component explains less than 1% of predictor variance but separates the two outcome groups perfectly in this constructed population. This is not a claim that the example represents a typical real data set; it shows why unsupervised variance retention is not a general criterion for predictive usefulness.

Interpreting PCA Retention for Prediction

A variance-retention rule answers a particular question: how much of the variation among the predictors is preserved by the selected components? A prediction task asks a different question: how well does a model using those components estimate the specified outcome? The answers may agree, but they need not.

When the outcome is modeled linearly, the term $\beta_j^2\lambda_j$ is a useful way to understand each component’s contribution to the variance of the fitted signal under the stated assumptions. A large $\lambda_j$ can be offset by a small $\beta_j$; a small $\lambda_j$ can be paired with a large $\beta_j$. With nonlinear relationships, predictive value may not be captured by this linear decomposition at all.

Keeping all components is a rotation of the predictor coordinates, not a reduction in the information represented by those predictors. In a full-rank linear model, using all components retains the same linear predictor space as using the original variables. Information is discarded when components are omitted; the fact that a component falls below a chosen variance threshold does not establish that the discarded information is irrelevant to the outcome.

State the goal.
Decide whether the task is predictor compression, reconstruction, or prediction of a particular outcome. A variance threshold directly addresses the first two, not the last.

Separate the quantities.
Report explained predictor variance as a PCA summary. Do not label it outcome variance explained or predictive accuracy.

Assess predictive relevance for the outcome.
Consider how each component relates to the target and whether a proposed model can represent that relationship. A small eigenvalue alone is not a reason to discard a component.

Check the prediction objective.
Choose components and modeling decisions with the outcome-specific prediction goal in mind. For a predictive task, the relevant evidence is performance for that task, not cumulative PCA variance by itself.

The final step is aligned with the SOA Exam PA (Predictive Analytics) syllabus: keep the unsupervised PCA summary distinct from the supervised prediction objective. A report can state both the percentage of predictor variance retained and the evidence about prediction, but should not treat one as a substitute for the other.

Common Exam Trap

A common error is to say, “The first two components explain 97.6% of predictor variance, so they must explain 97.6% of the outcome.” The conclusion does not follow. In the first worked example, the first two components explain $41/42=97.62\%$ of predictor variance, while the linear model’s outcome variance associated with those components is $169/178=94.94\%$. Those percentages have different denominators and describe different quantities.

A second trap is to justify adding variances by calling the component scores independent. PCA guarantees that distinct scores are uncorrelated, not generally independent. For a variance sum, state that the scores are uncorrelated and that the residual has zero covariance with them. Then the covariance terms vanish, which is the required justification.

Key takeaway: PCA orders components by predictor variance, not by predictive importance. For a linear outcome model with uncorrelated scores and an uncorrelated residual, a component’s signal contribution is $\beta_j^2\lambda_j$. A low-variance component can therefore matter for prediction, and a high-variance component can matter little.

Check Your Understanding

For each problem, keep predictor variance and outcome-related quantities separate. Assume population quantities unless the problem states otherwise.

Two uncorrelated scores have variances 16 and 1. A centered outcome is $Y=0.5T_1+3T_2+\varepsilon$, where $\varepsilon$ is uncorrelated with both scores and has variance 4. Calculate the percentage of predictor variance explained by $T_2$, its contribution to outcome signal variance, and the residual variance when only $T_1$ is used. (Answer: 5.88% of predictor variance; 9 units of signal variance; residual variance 13.)
Three uncorrelated scores have variances 25, 4, and 1. Their coefficients in a linear outcome model are 0.2, 1, and 5, respectively. Which score explains the least predictor variance, which contributes the most outcome signal variance, and what is that signal contribution? (Answer: $T_3$ explains the least predictor variance and contributes the most signal variance, $5^2(1)=25$.)
Two uncorrelated scores have variances 9 and 0.5, and the outcome model is $Y=T_1+2T_2+\varepsilon$, with residual variance 5 and zero covariance between the residual and both scores. Calculate total outcome variance and the mean squared error if only $T_1$ is retained. (Answer: total outcome variance 16; mean squared error 7.)
Four equally likely records have scores $(T_1,T_2)=(4,1),(4,-1),(-4,1),(-4,-1)$. The binary outcome is 1 exactly when $T_2=1$. Find the proportion of predictor variance explained by $T_2$, and compare the classification error achievable using only $T_1$ with that using $T_2$. (Answer: 5.88% of predictor variance; error using $T_1$ is 50%, while a threshold on $T_2$ has 0% error.)

What You'll Learn