Why the Choice of Matrix Matters
Principal components analysis (PCA) finds directions that summarize variation in a set of variables. Before interpreting a component, however, you must know which variation the analysis was designed to summarize. PCA based on a covariance matrix emphasizes variation in the variables’ original scales. PCA based on a correlation matrix first puts their scales on a comparable basis. These choices can produce different component directions and different conclusions.
Neither choice is universally correct. Covariance PCA is appropriate when the original measurement scales and their variances are meaningful to the analysis. Correlation PCA is often appropriate when variables use different units or when each variable should have comparable influence regardless of its original variability. The decision should reflect the question being asked, not a rule that one matrix is always superior. This distinction is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
What the Two Analyses Optimize
In covariance PCA, a unit-length weight vector \(w\) defines a component score as a linear combination of centered variables. Its sample variance is \(w^\mathsf{T}Sw\). The first component chooses \(w\) to maximize that variance subject to \(w^\mathsf{T}w=1\). Later components maximize variance subject to being orthogonal to the earlier component directions.
In correlation PCA, the same optimization is applied to variables measured in standard-deviation units. If \(Z\) denotes the centered variables divided by their respective sample standard deviations, then its sample covariance matrix is \(R\). Thus, correlation PCA maximizes the variance of combinations of these rescaled variables. Its components describe variation relative to each variable’s own spread, rather than variation in the original measurement units.
Here \(X_c\) is the centered data matrix. The first expression applies to covariance PCA; the second gives the matrix used for correlation PCA. A correlation is unit-free and lies between \(-1\) and \(1\), while a covariance retains the product of the two variables’ measurement units.
Why Correlation PCA Uses the Rescaled Covariance Matrix
The connection between the two methods follows directly from how sample covariance is calculated. Let \(Z=X_cD^{-1}\), so each column of \(X_c\) is divided by its sample standard deviation. Since the sample covariance matrix of a centered data matrix \(A\) is \(A^\mathsf{T}A/(n-1)\), the covariance matrix of \(Z\) is:
The equality uses the fact that \(D\) is diagonal and symmetric, so \(D^{-1}\) can be placed on either side of \(X_c^\mathsf{T}X_c\) as shown. This establishes why an eigenanalysis of \(R\) is the covariance PCA of the rescaled columns. The rescaling changes the question: each variable is expressed relative to its own sample standard deviation.
Worked Comparisons
Worked Example: A Large Scale Dominates Covariance PCA
A hypothetical risk study records an annual dollar amount \(X\) and a dimensionless rate \(Y\). Suppose their sample standard deviations are \(10\) and \(1\), respectively, and their sample correlation is \(0.6\). Compare the first principal component from covariance PCA with the first component from correlation PCA. Use the given sample summaries.
Method. First recover the covariance, then find the eigenvalues and leading direction for each matrix. This avoids treating the two differently scaled variables as though their covariance matrix were already scale-neutral.
The covariance is \(0.6(10)(1)=6\). The sample covariance matrix and corresponding correlation matrix are therefore:
For \(S\), the trace is \(101\) and the determinant is \(100(1)-6^2=64\). Its eigenvalues solve \(\lambda^2-101\lambda+64=0\), giving:
For the leading eigenvalue \(\lambda_1\), the first row of \((S-\lambda_1I)w=0\) gives \(6w_2=(\lambda_1-100)w_1\). Taking \(w_1=1\), the direction is proportional to \((1,0.060385)^\mathsf{T}\). Normalizing gives approximately \((0.9982,0.0603)^\mathsf{T}\). The share of total covariance variance captured is \(100.3623/101\approx0.9937\), or \(99.37\%\).
For \(R\), the eigenvalues are \(1.6\) and \(0.4\). The leading eigenvector is \((1,1)^\mathsf{T}/\sqrt{2}\), approximately \((0.7071,0.7071)^\mathsf{T}\). Its share of total correlation-matrix variance is \(1.6/2=80\%\).
Check. Each pair of eigenvalues sums to the trace of its matrix: \(100.3623+0.6377=101\) and \(1.6+0.4=2\). Covariance PCA places nearly all its weight on \(X\), whose variance is much larger. Correlation PCA gives equal weights to the two rescaled variables. The common mistake is to infer from the \(99.37\%\) figure that covariance PCA has captured nearly all variation on every meaningful scale; that percentage refers to total variance in the original-scale variables.
Worked Example: Choosing a Matrix for a Monitoring Goal
A hypothetical portfolio monitoring team tracks an annual loss amount \(A\), measured in thousands of dollars, and a unit-free claim rate \(B\). Its sample covariance matrix is:
The team wants a single summary of overall variation in the recorded original-scale measurements. Find the leading covariance component and explain whether correlation PCA would answer the same question.
Method. Use covariance PCA because the stated goal is to summarize variation on the recorded scales. The standard deviations are \(\sqrt{900}=30\) and \(\sqrt{4}=2\). Therefore the correlation is \(18/(30\cdot2)=0.3\), and:
For \(S\), the trace is \(904\) and determinant is \(900(4)-18^2=3276\). The eigenvalues are:
For the leading eigenvalue, \(18w_2=(\lambda_1-900)w_1\). Taking \(w_1=1\), the direction is proportional to \((1,0.02008)^\mathsf{T}\); after normalization it is approximately \((0.9998,0.0201)^\mathsf{T}\). The leading component explains \(900.3615/904\approx99.60\%\) of the total covariance variance.
For correlation PCA, the eigenvalues of \(R\) are \(1.3\) and \(0.7\), with leading direction \((1,1)^\mathsf{T}/\sqrt{2}\). It explains \(1.3/2=65\%\) of the total correlation variance. That analysis gives the two variables equal influence in standard-deviation units, which is not the stated goal of summarizing original-scale variation.
Check. The covariance eigenvalues sum to \(904\), and the correlation eigenvalues sum to \(2\). The covariance result is dominated by the loss amount’s much larger variance; correlation PCA would instead answer a relative-variation question. The common mistake is to choose correlation PCA automatically whenever variables have different units, without checking whether those original-scale differences are substantively important to the stated objective.
Worked Example: Equal Variances Give the Same Directions
Suppose two variables have sample covariance matrix \(S=\begin{pmatrix}4&2\\2&4\end{pmatrix}\). Determine whether covariance and correlation PCA give the same leading direction, and calculate the leading variance share under each method.
Method. Compare the standard deviations first. Both are \(\sqrt{4}=2\), so the correlation matrix is \(S\) divided by \(2\cdot2=4\):
For \(S\), the vectors \((1,1)^\mathsf{T}\) and \((1,-1)^\mathsf{T}\) have eigenvalues \(6\) and \(2\), respectively. Thus the leading direction is \((1,1)^\mathsf{T}/\sqrt{2}\), and its variance share is \(6/(6+2)=75\%\). For \(R\), the same directions have eigenvalues \(1.5\) and \(0.5\); the leading direction is unchanged and its share is \(1.5/(1.5+0.5)=75\%\).
Check. Both eigenvalue pairs sum to their matrix traces, \(8\) and \(2\). Here the equal standard deviations make the matrices scalar multiples of each other, so their eigenvectors and variance shares agree. This is a special case, not a reason to assume the methods always agree.
Checking Eigenvector Normalization
An eigenvector gives a direction, so multiplying all its entries by the same nonzero constant does not change the component direction. For reporting weights, it is standard to normalize the vector to have length one. For example, suppose a numerical eigensolver returns the dominant direction \((1,0.1813346)^\mathsf{T}\). Its length is:
Dividing each entry by this length gives approximately \((0.9840,0.1784)^\mathsf{T}\), to four decimal places. The first coordinate is \(1/\sqrt{1+0.1813346^2}\approx0.9839537\), which rounds to \(0.9840\). A useful check is that the squared normalized entries sum to approximately one.
Making and Explaining the Choice
A defensible choice starts with the analysis goal and the meanings of the variables. Ask whether the goal concerns total variation on the original scales or whether variables should contribute on a comparable relative scale. Then inspect their units and sample variances. If the scales differ substantially, covariance PCA may be driven mainly by the variable with the largest variance. That may be exactly right when large-scale variation is the intended signal; it may be misleading when the variables are meant to be treated comparably.
| Consideration | Covariance PCA | Correlation PCA |
|---|---|---|
| Input matrix | Sample covariance matrix \(S\) | Sample correlation matrix \(R\) |
| Scale represented | Original measurement scales | Standard-deviation units |
| Typical reason to choose | Original variances are meaningful to the goal | Variables have different units or should be comparable in influence |
| Key interpretation | Variance share is relative to total original-scale variance | Variance share is relative to total correlation-matrix variance |
A zero-variance variable requires special attention: its sample standard deviation is zero, so its correlations are undefined and it cannot be included in \(R\) as defined above. Also, a correlation matrix does not make all variables equally important in every substantive sense. It gives each variable unit variance in the analysis, but the analyst must still justify why that treatment fits the question.
Check Your Understanding
For each problem, decide what the calculations imply about the matrix choice or the PCA result.
- Two variables have sample standard deviations \(5\) and \(1\), with sample correlation \(-0.4\). What is their sample covariance? (Answer: \(-2\))
- A two-variable sample covariance matrix is \(\begin{pmatrix}16&0\\0&1\end{pmatrix}\). What is the correlation matrix, and is its leading direction unique? (Answer: \(R=I_2\); the eigenvalues are tied, so there is no unique leading direction.)
- A covariance matrix is \(\begin{pmatrix}9&3\\3&9\end{pmatrix}\). Find the leading eigenvalue and the proportion of total covariance variance it explains. (Answer: Leading eigenvalue \(12\); proportion \(12/18=2/3\), or \(66.67\%\).)
- Two features have standard deviations \(40\) and \(2\), with correlation \(0.25\). Find their covariance and state which matrix is a reasonable starting point if the goal is to give each feature comparable influence despite its original scale. (Answer: Covariance \(=0.25(40)(2)=20\); correlation PCA is reasonable for comparable influence.)
- A two-variable covariance matrix is \(\begin{pmatrix}25&10\\10&16\end{pmatrix}\). Find the correlation and the leading direction of the correlation matrix. (Answer: Correlation \(10/(5\cdot4)=0.5\); leading correlation-PCA direction is \((1,1)^\mathsf{T}/\sqrt{2}\).)