Why Replace Predictors with Principal Components?
A predictive analytics data set may contain several numeric variables that measure nearly the same underlying characteristic. For example, a hypothetical insurance portfolio might record both annual mileage and annual fuel use for each vehicle. Keeping both variables can make the feature set redundant. Principal components analysis (PCA) offers a way to replace such variables with new, uncorrelated numeric features that summarize variation in the original measurements.
The purpose of PCA is compression of variation. It finds linear combinations that capture as much variation in the input variables as possible, in a ranked sequence. A small number of components can therefore represent much of the variation in a larger set of numeric variables. This can be useful when creating features for a later predictive model, exploring correlated measurements, or reducing the dimension of a data set.
That purpose is narrower than it may first appear. PCA does not use a response variable to identify useful predictors, and it does not explain why the variables move together. A component that captures little variation can still be important for predicting an outcome; a component that captures much variation need not predict the outcome at all.
The \(p\) directions give a new coordinate system for describing the same observations. Keeping all \(p\) components changes the coordinates but does not reduce the number of features. Keeping only the first \(k\) components, where \(k<p\), is the dimension-reduction step: it discards the variation in the omitted directions. PCA can therefore compress a data set, but compression necessarily loses some information unless the omitted components have zero variance.
PCA applies to numeric variables. Categorical variables are not automatically suitable inputs just because they have been stored as numbers; a numerical code may be only a label. The choice between using raw measurement scales and standardized scales also matters, because variables with larger numerical variance can dominate a covariance-based analysis. Those are data-preparation decisions, not evidence that PCA has discovered a more important business concept.
Variance Is the Criterion, Not the Outcome
Suppose the input variables have been centered, and let \(\Sigma\) denote their covariance matrix. Centering means subtracting each variable's mean from its observations; it makes the component scores describe variation around the mean rather than the location of the data. For a direction \(a\), the variance of its component score is
PCA restricts directions to unit length so that multiplying a direction by an arbitrarily large constant cannot artificially inflate its score variance. It then chooses the direction with the greatest score variance. Later directions are constrained to be orthogonal to earlier directions. This criterion does not involve a response \(Y\), a prediction error, or a causal model.
The variance formula follows from the definition of covariance. Write the score as \(T=\sum_{r=1}^{p}a_rX_r\). Variance of a sum is the sum of all pairwise covariances weighted by the coefficients, so
This also explains why direction matters: a linear combination can have a different variance from any one input variable because its variance depends on both the individual variances and the covariances. If two scores have orthogonal PCA directions, their covariance is zero under the covariance-matrix construction. Uncorrelated does not generally mean independent; independence requires additional conditions and does not follow from PCA alone.
A common summary is the proportion of total variance represented by the retained components. If the component variances are \(\lambda_1,\ldots,\lambda_p\), with \(\lambda_1\geq\cdots\geq\lambda_p\), then the first \(k\) components represent the proportion
This is a descriptive summary of variation in the PCA inputs. It is not the proportion of outcome variation explained by a predictive model, nor does it report prediction accuracy.
Worked Example: Compressing Two Correlated Measurements
A hypothetical vehicle study has two centered numeric measurements: annual mileage \(X_1\) and annual fuel use \(X_2\). Their covariance matrix is \(\begin{pmatrix}9&6\\6&9\end{pmatrix}\). The component directions have already been determined as \(a_1=(1,1)^\mathsf{T}/\sqrt{2}\) and \(a_2=(1,-1)^\mathsf{T}/\sqrt{2}\). Find the variance represented by the first component, its share of total variance, and the component scores and one-component reconstruction for an observation with \(X_1=2\), \(X_2=0\).
Method. Use the supplied directions to form component scores, calculate their variances from the covariance matrix, and compare the first score with the original coordinates. The directions are supplied so the calculation focuses on what dimension reduction means, rather than on finding the directions.
For the first direction, the score is \(T_1=(X_1+X_2)/\sqrt{2}\). Its variance is \[ a_1^\mathsf{T}\Sigma a_1 =\frac{1}{2}(9+6+6+9)=15. \] For the second direction, \(T_2=(X_1-X_2)/\sqrt{2}\), and its variance is \[ a_2^\mathsf{T}\Sigma a_2 =\frac{1}{2}(9-6-6+9)=3. \] The total input variance is \(9+9=18\), which also equals \(15+3\). Thus the first component represents \(15/18=0.8333\), or \(83.33\%\), of the total variance, rounded to two decimal places.
For the stated observation, \(T_1=(2+0)/\sqrt{2}=\sqrt{2}\) and \(T_2=(2-0)/\sqrt{2}=\sqrt{2}\). Keeping only the first component and projecting back onto its direction gives \[ T_1a_1=\sqrt{2}\frac{(1,1)^\mathsf{T}}{\sqrt{2}}=(1,1)^\mathsf{T}. \] The original vector is \((2,0)^\mathsf{T}\), so the discarded difference is \((1,-1)^\mathsf{T}\). The reconstruction retains the common movement in the two measurements but loses their contrast for this observation.
Check. The two component variances sum to \(18\), the trace of the covariance matrix. The first component's share is \(15/18\), and adding the omitted component would recover the original centered observation exactly. A common mistake is to say the first component preserves \(83.33\%\) of predictive accuracy; the calculation establishes only the share of input variance.
High Variance Does Not Mean High Predictive Value
PCA is unsupervised: the component directions are selected without looking at the target. This distinction matters in predictive analytics. A large source of variation in the predictors may be unrelated to the outcome, while a relatively small source of variation may carry a strong outcome signal. If only the first few components are retained, that signal can be discarded.
Worked Example: A Low-Variance Component Carries the Signal
In a hypothetical claims model, two already-constructed component scores \(T_1\) and \(T_2\) are centered and uncorrelated, with \(\operatorname{Var}(T_1)=100\) and \(\operatorname{Var}(T_2)=4\). The response is \(Y=T_2+\varepsilon\), where \(\varepsilon\) is independent of both scores, has mean zero, and has variance \(1\). Compare the variance retained by keeping only \(T_1\) with the predictive information in \(T_2\).
Method. First calculate the total variance of the two input scores and the share due to \(T_1\). Then calculate the variance of \(Y\) and the fraction explained by the linear predictor \(T_2\). This separates PCA's variance criterion from response prediction.
The total predictor variance is \(100+4=104\), so retaining only \(T_1\) retains \(100/104=0.9615\), or \(96.15\%\), rounded to two decimal places. However, \(T_1\) has no covariance with \(Y\): \[ \operatorname{Cov}(T_1,Y) =\operatorname{Cov}(T_1,T_2+\varepsilon)=0+0=0. \] The response variance is \(\operatorname{Var}(Y)=4+1=5\). Using \(T_2\) as the predictor gives \(Y\)'s signal component exactly, leaving only \(\varepsilon\); the fraction of response variance accounted for is \(4/5=0.8000\).
Thus a PCA rule retaining only the high-variance direction would keep \(96.15\%\) of predictor variance but discard the component that accounts for \(80.00\%\) of response variance. Check. The calculations use the specified independence and zero-mean assumptions: without them, the variance and covariance results need not hold. The exam trap is to treat a large retained-variance percentage as proof of good prediction; it is not.
Components Are Not Causal Explanations
A component describes a direction of joint variation in the measured variables. Its coefficients can help summarize which variables move together, but they do not establish that one variable causes another. The same observed covariance structure can arise from different underlying mechanisms.
Worked Example: One Covariance Pattern, Different Explanations
Consider two centered measurements, \(A\) and \(B\), with variance \(1\) each and covariance \(0.8\). First summarize their shared and contrast directions. Then compare two hypothetical data-generating accounts that produce this same covariance information.
Method. Use the common direction \((1,1)^\mathsf{T}/\sqrt{2}\) and contrast direction \((1,-1)^\mathsf{T}/\sqrt{2}\) to describe variation. Then check the means, variances, and covariance under each proposed account. These calculations illustrate why a PCA pattern is not a causal identification method.
The covariance matrix is \(\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\). The common score \((A+B)/\sqrt{2}\) has variance \((1+0.8+0.8+1)/2=1.8\); the contrast score \((A-B)/\sqrt{2}\) has variance \((1-0.8-0.8+1)/2=0.2\). Total variance is \(2\), so the common direction represents \(1.8/2=90\%\).
One possible account is a shared factor: let \(A=U+e_A\) and \(B=U+e_B\), where the independent, mean-zero terms \(U,e_A,e_B\) have variances \(0.8,0.2,0.2\). Then each observed variance is \(0.8+0.2=1\), and \(\operatorname{Cov}(A,B)=\operatorname{Var}(U)=0.8\).
A different account is a directed relationship: let \(A\) have mean zero and variance \(1\), and let \(B=0.8A+e\), where \(e\) is independent of \(A\), has mean zero, and has variance \(0.36\). Then \(\operatorname{Cov}(A,B)=0.8\operatorname{Var}(A)=0.8\), and \(\operatorname{Var}(B)=0.8^2(1)+0.36=1\). Both accounts produce the same means and covariance matrix, and therefore the same PCA variance summary.
Check. The shared-factor account and the directed account have different causal interpretations despite the same covariance pattern. PCA identifies neither account; additional design information and appropriate causal analysis would be needed. The trap is to label a component such as “shared risk” and then claim it proves a common cause.
Using PCA Responsibly for Feature Creation
PCA is useful when a smaller set of numeric features is desirable and the main objective is to summarize variation among the inputs. It may reduce redundancy, create orthogonal features, or make a high-dimensional set easier to inspect. Its components are weighted combinations of original variables, so interpretation depends on the variables included, their units, and the coefficients defining each direction.
For a predictive task, the component count should not be chosen solely because a retained-variance threshold looks large. The relevant question is whether the reduced feature set supports the intended prediction task. A low-variance component may carry predictive signal, as the preceding example showed. Conversely, retaining a component because it has high variance does not make its variation relevant to the response.
PCA also does not automatically solve every modeling problem. It produces linear combinations, so it may not capture nonlinear structure. Orthogonal component scores are uncorrelated under the stated construction, but they are not guaranteed to be independent or easier to explain to stakeholders. A component may combine several original measurements in a way that is statistically concise but operationally difficult to interpret.
- Ask what is being summarized. The variance proportions refer to the selected input variables, not to the response.
- Check the measurement scales. A variable's units and variance can influence the directions; a large numerical scale is not automatically more important.
- Separate prediction from explanation. Evaluate predictive usefulness using the response and an appropriate modeling process; do not infer it from PCA variance alone.
- Avoid causal language. Components describe patterns of variation, not mechanisms that generated those patterns.
Check Your Understanding
For each problem, distinguish what PCA summarizes from what would require a predictive or causal analysis.
- A hypothetical two-variable data set has component variances \(12\) and \(3\). Calculate the share of total variance represented by the first component. State what this percentage does not establish. (Answer: \(12/(12+3)=0.8000\), or \(80\%\); it does not establish predictive accuracy or causal importance.)
- A team retains components representing \(45\) units of variance out of \(60\) total units. Calculate the retained-variance proportion and state whether that alone proves the reduced features predict well. (Answer: \(45/60=0.7500\), or \(75\%\); no, predictive performance requires assessment against the response.)
- Two uncorrelated component scores have variances \(25\) and \(1\). A response is \(Y=T_2+\varepsilon\), where \(\varepsilon\) is independent of both scores and has variance \(4\). Calculate the proportion of predictor variance in \(T_1\) and the fraction of response variance represented by \(T_2\). (Answer: \(25/(25+1)=0.9615\), or \(96.15\%\), in \(T_1\); \(\operatorname{Var}(Y)=1+4=5\), so \(T_2\) represents \(1/5=0.2000\), or \(20\%\), of response variance.)
- A PCA of two centered measurements yields a common-direction component with \(90\%\) of their total variance. Can this result establish that the measurements share a causal driver? Explain what additional information is needed. (Answer: No; the component summarizes shared variation, but different causal mechanisms can produce the same covariance pattern. Additional design information and causal analysis are needed.)
- A numeric variable is recorded in dollars while a related variable is recorded in thousands of dollars. Before interpreting a PCA component, what issue should the analyst investigate, and why? (Answer: Investigate whether the variables should be put on comparable scales; their units and variances can affect the PCA directions, so raw numerical scale should not be mistaken for substantive importance.)