Covariance PCA Versus Correlation PCA

Two Ways to Define Principal Components

PCA can be applied either to the covariance matrix of the observed variables or to their correlation matrix. These choices need not produce the same principal directions or the same proportions of variance explained. The distinction arises because correlation PCA first places variables on a common variance scale, whereas covariance PCA uses their original variances.

Let $X=(X_1,\ldots,X_p)^\mathsf{T}$ have mean vector $\mu$ and positive, finite variances. Write $\Sigma=\operatorname{Cov}(X)$, and let $D$ be the diagonal matrix containing those variances. The correlation matrix is obtained by standardizing each variable by its standard deviation. We assume throughout that every diagonal entry of $D$ is strictly positive, so this standardization is defined.

Definition: Covariance PCA uses the eigenvectors and eigenvalues of $\Sigma$. Correlation PCA uses the eigenvectors and eigenvalues of $R=D^{-1/2}\Sigma D^{-1/2}$, the correlation matrix of the variables. For either matrix, the proportion of total variance explained by its $j$th principal component is its $j$th eigenvalue divided by the trace of the matrix.

If $q$ is a unit eigenvector of $R$ with eigenvalue $\lambda$, the corresponding correlation-PCA score is $q^\mathsf{T}D^{-1/2}(X-\mu)$. Its coefficients on the standardized variables are the entries of $q$. Covariance PCA instead chooses a unit eigenvector $a$ of $\Sigma$ and uses the score $a^\mathsf{T}(X-\mu)$. The vectors $a$ and $q$ therefore describe coefficients in different coordinates and under different constraints.

How the Matrices and Variance Proportions Relate

The two matrices are related by

\Sigma=D^{1/2}RD^{1/2}.

This is a congruence transformation, not generally a similarity transformation. A similarity transformation preserves eigenvalues; a congruence transformation of this form does not generally preserve either eigenvalues or eigenvectors. The total covariance variance is $\operatorname{tr}(\Sigma)=\sum_{j=1}^p\operatorname{Var}(X_j)$, while $\operatorname{tr}(R)=p$. Thus the two variance-proportion calculations use different totals.

The relationship can also be described as an optimization problem. For any unit vector $b$, the variance of the linear score $b^\mathsf{T}(X-\mu)$ is $b^\mathsf{T}\Sigma b$; covariance PCA maximizes this quantity subject to $b^\mathsf{T}b=1$. Correlation PCA maximizes the variance of a score formed from standardized variables, subject to a unit-length constraint on its coefficients in standardized coordinates.

Result: If $q$ is a unit eigenvector of $R$ with eigenvalue $\lambda$, then $a=D^{-1/2}q$ satisfies $$ \Sigma a=\lambda D a,\qquad a^\mathsf{T}Da=1. $$ Thus correlation PCA can be expressed in the original coordinates as a generalized eigenvalue problem, with normalization determined by $D$, rather than the ordinary unit-length normalization used in covariance PCA.

To verify this result, use $\Sigma=D^{1/2}RD^{1/2}$ and $a=D^{-1/2}q$:

\Sigma a =D^{1/2}RD^{1/2}D^{-1/2}q =D^{1/2}Rq =\lambda D^{1/2}q =\lambda Da.

Also, $a^\mathsf{T}Da=q^\mathsf{T}D^{-1/2}DD^{-1/2}q=q^\mathsf{T}q=1$. This shows precisely how the eigenproblem and normalization change. It does not imply that the covariance-PCA eigenvector is $D^{-1/2}q$: that vector solves the generalized problem, not in general the ordinary eigenproblem for $\Sigma$.

Worked Comparisons

Worked Example: A Direction Changes After Standardization

Suppose the covariance matrix is $$ \Sigma=\begin{pmatrix}9&6\\6&4\end{pmatrix}. $$ The variable standard deviations are $3$ and $2$, so $$ D=\begin{pmatrix}9&0\\0&4\end{pmatrix},\qquad R=D^{-1/2}\Sigma D^{-1/2} =\begin{pmatrix}1&1\\1&1\end{pmatrix}. $$

For covariance PCA, the characteristic polynomial is $\det(\Sigma-\lambda I)=(9-\lambda)(4-\lambda)-36=\lambda(\lambda-13)$. The leading eigenvalue is $13$. Solving $(\Sigma-13I)a=0$ gives $a_2=(2/3)a_1$, so a unit leading eigenvector is $a=(3,2)^\mathsf{T}/\sqrt{13}$. The other eigenvalue is $0$. The covariance variance proportions are therefore $13/13=1$ and $0/13=0$.

For correlation PCA, the eigenvectors of $R$ are $(1,1)^\mathsf{T}/\sqrt{2}$ and $(1,-1)^\mathsf{T}/\sqrt{2}$, with eigenvalues $2$ and $0$. The correlation-PCA proportions are $2/2=1$ and $0/2=0$. Both matrices describe perfect positive linear dependence, but their leading coefficient vectors differ: covariance PCA uses $(3,2)^\mathsf{T}/\sqrt{13}$, while correlation PCA uses equal coefficients on the standardized variables.

Worked Example: Variance Proportions Can Differ Substantially

Consider $$ \Sigma=\begin{pmatrix}100&9\\9&1\end{pmatrix}. $$ Its determinant is $100-81=19>0$, and its leading diagonal entry is positive, so this is positive definite. The standard deviations are $10$ and $1$; hence the correlation is $9/(10\cdot1)=0.9$, giving $$ R=\begin{pmatrix}1&0.9\\0.9&1\end{pmatrix}. $$

The eigenvalues of $\Sigma$ solve $\lambda^2-101\lambda+19=0$. They are $$ \lambda_1=\frac{101+45\sqrt{5}}{2}\approx100.8115,\qquad \lambda_2=\frac{101-45\sqrt{5}}{2}\approx0.1885. $$ Their sum is $101=\operatorname{tr}(\Sigma)$. The covariance-PCA proportions are therefore approximately $100.8115/101=0.9981$ and $0.1885/101=0.0019$.

For $R$, the eigenvectors are proportional to $(1,1)^\mathsf{T}$ and $(1,-1)^\mathsf{T}$, with eigenvalues $1.9$ and $0.1$. Since the trace is $2$, the correlation-PCA proportions are $1.9/2=0.95$ and $0.1/2=0.05$. The leading covariance eigenvector has coefficient ratio $$ \frac{a_2}{a_1}=\frac{\lambda_1-100}{9} =\frac{45\sqrt{5}-99}{18}\approx0.0902, $$ whereas the leading correlation eigenvector has equal coefficients on standardized variables. The leading directions differ, as do the variance proportions.

Worked Example: Equal Variances Give the Same Directions

Let $$ \Sigma=\begin{pmatrix}4&2\\2&4\end{pmatrix}. $$ Both variables have variance $4$, so $D=4I$ and $$ R=D^{-1/2}\Sigma D^{-1/2}=\frac14\Sigma =\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}. $$

The shared eigenvectors are $(1,1)^\mathsf{T}/\sqrt{2}$ and $(1,-1)^\mathsf{T}/\sqrt{2}$. The covariance eigenvalues are $6$ and $2$, with proportions $6/8=0.75$ and $2/8=0.25$. The correlation eigenvalues are $1.5$ and $0.5$, with proportions $1.5/2=0.75$ and $0.5/2=0.25$. In this case, standardization changes the eigenvalues by a common factor but changes neither the directions nor the proportions.

When Do the Two PCAs Agree?

The equal-variance example is a special case of a general result. If every variable has the same positive variance $c$, then $D=cI$, so $R=\Sigma/c$. Multiplication of a matrix by a positive scalar leaves its eigenvectors unchanged and multiplies each eigenvalue by that scalar. Consequently the variance proportions also remain unchanged.

Theorem: If $D=cI$ for some $c>0$, covariance PCA and correlation PCA have the same eigenspaces and the same variance proportions. Their eigenvalues differ by the factor $c$.

Proof. From $D=cI$, $$ R=D^{-1/2}\Sigma D^{-1/2}=\frac{1}{c}\Sigma. $$ If $\Sigma v=\lambda v$, then $Rv=(\lambda/c)v$, so $v$ is also an eigenvector of $R$. Conversely, an eigenvector of $R$ is an eigenvector of $\Sigma=cR$. The covariance eigenvalues sum to $\operatorname{tr}(\Sigma)$, and the corresponding correlation eigenvalues sum to $\operatorname{tr}(\Sigma)/c$. Each eigenvalue and the total are divided by $c$, leaving each eigenvalue-to-total ratio unchanged. This proves the claim, including the eigenspace statement when eigenvalues are repeated. $\square$

Outside this special case, there is no general equality of eigenvectors or proportions. Unequal marginal variances alter the matrix entries through $D^{-1/2}$, and the resulting change can affect both the leading direction and how total variance is distributed across components. The examples show that the change is not merely a rescaling of every eigenvalue by one common constant.

Sample Matrices and a Common Pitfall

For a sample of $n$ observations, let $S$ be the sample covariance matrix and let $D_s$ contain its diagonal variances. When each sample variance is positive, the sample correlation matrix is $R_s=D_s^{-1/2}SD_s^{-1/2}$. The comparison follows the same algebra as for population matrices. A common positive divisor used for every entry of $S$ does not affect its eigenvectors or its variance proportions, but replacing $S$ by $R_s$ is generally not such a common scalar multiplication.

Wrong reasoning: “A correlation matrix is just the covariance matrix on a different numerical scale, so its PCA directions must be the same.”
Correction: The entries are rescaled according to the marginal standard deviations. Unless those variances are all equal, $R_s$ is not generally a scalar multiple of $S$; its eigenvectors and variance proportions can differ.
Another common error: comparing the largest covariance eigenvalue directly with the largest correlation eigenvalue as though they were in the same units. Each variance proportion must use the total variance for its own matrix.

Neither approach is automatically preferable in every analysis. They answer different variance-maximization questions: covariance PCA uses the variables in their original coordinates, while correlation PCA gives each standardized variable unit variance before forming scores. A principled interpretation must therefore state which matrix was analyzed and which coordinates the reported coefficients refer to.

Check Your Understanding

Use the definitions and calculations above to check your understanding of the comparison.

For positive marginal variances, write the relationship between $\Sigma$, $R$, and $D$.
Why does the fact that $R=D^{-1/2}\Sigma D^{-1/2}$ not generally guarantee that the two matrices have the same eigenvectors?
If every variable has variance $c>0$, how do the eigenvalues of $R$ relate to those of $\Sigma$, and what happens to the variance proportions?
What generalized eigenvalue equation does a correlation-PCA coefficient vector in the original coordinates satisfy?
In the second worked example, why is the first covariance-PCA proportion so much larger than the first correlation-PCA proportion?