Why Calculate PCA From a Correlation Matrix?
Principal component analysis (PCA) finds directions that summarize variation across several variables. The matrix used for PCA matters: a covariance matrix describes variation on the variables’ original scales, while a correlation matrix describes variation after each variable has been scaled to unit variance. Here the focus is the second approach: standardize each variable, then calculate PCA from its correlation matrix.
This is useful when variables are measured in different units or have very different numerical scales. For example, a data set might contain an account balance in dollars and a claim frequency in counts. Standardizing makes each variable’s sample variance equal to one before the component directions are found. It does not make the variables uncorrelated; their pairwise correlations remain the information used to determine the directions.
The standardization uses the sample standard deviation, not the sample variance. Each standardized variable has sample mean zero and sample variance one. The diagonal entries of \(R\) are therefore one; its off-diagonal entries are the sample correlations, each between \(-1\) and \(1\). The matrix is symmetric.
For correlation-based PCA, find the eigenvalues and orthonormal eigenvectors of \(R\). The eigenvectors give directions in standardized-variable coordinates. The eigenvalues give the variances of the corresponding component scores. If \(v_k\) is an eigenvector and \(z_i\) is the row vector of standardized values for observation \(i\), its score on component \(k\) is \(z_i v_k\).
Why Standardization Produces the Correlation Matrix
Let \(Z\) be the \(n\)-by-\(p\) matrix whose \(j\)th column contains the standardized observations for variable \(j\). Each column is centered. The sample covariance matrix of \(Z\) is \(Z^{\mathsf T}Z/(n-1)\). For columns \(j\) and \(k\), its entry is
The first equality substitutes the definition of each standardized value; the second is the sample correlation formula. Thus \(Z^{\mathsf T}Z/(n-1)=R\). PCA of the standardized data matrix and PCA of its correlation matrix therefore use the same eigenvalue problem. This identity requires that each included variable have positive sample standard deviation and that the same observations are used to calculate the means, standard deviations, and correlations.
A unit-length direction \(a\), with \(a^{\mathsf T}a=1\), gives standardized component scores \(Za\). Their sample variance is
Consequently, maximizing the variance of a standardized component is equivalent to maximizing \(a^{\mathsf T}Ra\) over unit-length directions. The eigenvector associated with the largest eigenvalue gives the first principal component direction. Subsequent orthogonal eigenvectors give the later directions. This is the same PCA optimization principle applied to standardized observations rather than to the original-scale observations.
Finding Directions From a Correlation Matrix
A correlation matrix is symmetric, so its eigenvectors can be chosen orthonormal. In practice, calculate or obtain the eigenvalues and eigenvectors, order the eigenvalues from largest to smallest, and use the corresponding eigenvectors as the component directions. A direction’s coefficients multiply the standardized variables—not the original measurements.
Worked Example: Eigenanalysis of a Three-Variable Correlation Matrix
A hypothetical portfolio study has three standardized indicators: \(Z_1\), \(Z_2\), and \(Z_3\). Its sample correlation matrix is
Find the component directions and eigenvalues. Identify which direction has the greatest variance.
Method. Solve \(Rv=\lambda v\). The first two variables form a \(2\)-by-\(2\) block, and the third variable is uncorrelated in this sample with the first two. For the first block, \[ \det\begin{pmatrix}1-\lambda&0.6\\0.6&1-\lambda\end{pmatrix} =(1-\lambda)^2-0.36=0. \] Thus \(1-\lambda=0.6\) or \(1-\lambda=-0.6\), giving eigenvalues \(0.4\) and \(1.6\). The third coordinate direction has eigenvalue \(1\).
For \(\lambda=1.6\), the first two coordinates must be equal, so a unit eigenvector is \(v_1=(1,1,0)^{\mathsf T}/\sqrt{2}\). For \(\lambda=0.4\), they must be opposites, giving \(v_3=(1,-1,0)^{\mathsf T}/\sqrt{2}\). For \(\lambda=1\), a unit eigenvector is \(v_2=(0,0,1)^{\mathsf T}\). These vectors are mutually orthogonal and each has length one.
The eigenvalues in descending order are \(1.6,1,0.4\), with the directions \(v_1,v_2,v_3\), respectively. The first component has the greatest variance. Its score for an observation is \((z_1+z_2)/\sqrt{2}\): it combines the two positively correlated indicators in the same direction. As a check, multiplying the matrix by \(v_1\) gives \(Rv_1=1.6v_1\), as required.
Common mistake avoided: using the original, unstandardized measurements as the coordinates in the score formula. These eigenvectors were calculated from \(R\), so the inputs to the score formula must be standardized values.
Building the Correlation Matrix From Observations
When raw observations are supplied, standardize each column before applying PCA from a correlation matrix. For hand calculations, it is often efficient to center each column, compute its sample standard deviation, and then evaluate the correlations using centered cross-products. The following example shows that the units disappear from those correlations, while the relationships between the variables remain.
Worked Example: Standardizing Variables on Very Different Scales
A hypothetical analyst records three indicators for four accounts. The data are shown below. Find the sample correlation matrix by standardizing the columns.
| Account | \(X_1\) (dollars) | \(X_2\) (index points) | \(X_3\) (thousandths) |
|---|---|---|---|
| 1 | 90 | 472 | 0.0086 |
| 2 | 90 | 504 | 0.0102 |
| 3 | 110 | 496 | 0.0114 |
| 4 | 110 | 528 | 0.0098 |
Step 1: Calculate means and centered values. The column means are \(\bar{x}_1=100\), \(\bar{x}_2=500\), and \(\bar{x}_3=0.0100\). The centered columns are
Step 2: Calculate sample standard deviations. The sums of squared centered values are \(400\), \(1600\), and \(0.000004\), respectively. Dividing by \(n-1=3\) gives sample variances \(400/3\), \(1600/3\), and \(0.000004/3\). Thus the sample standard deviations are \(20/\sqrt{3}\), \(40/\sqrt{3}\), and \(0.002/\sqrt{3}\).
Step 3: Express the standardized columns simply. Define \(a=(-1,-1,1,1)\), \(b=(-1,1,-1,1)\), and \(c=(-1,1,1,-1)\). Each vector has sum zero and squared length \(4\); their pairwise dot products are zero. Dividing the centered columns by their sample standard deviations gives
Step 4: Compute the correlations. Each diagonal entry is \(z_j^{\mathsf T}z_j/3=1\). For the first and second variables, \(z_1^{\mathsf T}z_2=(3/4)(0.6)(4)=1.8\), so \(r_{12}=1.8/3=0.6\). Similarly, \(z_1^{\mathsf T}z_3=1.8\), so \(r_{13}=0.6\). Finally, orthogonality of \(a,b,c\) gives \(z_2^{\mathsf T}z_3=(3/4)(0.6)(0.6)(4)=1.08\), so \(r_{23}=1.08/3=0.36\). Therefore,
Check. The matrix is symmetric, its diagonal entries are one, and each off-diagonal entry lies between \(-1\) and \(1\). The units and sizes of the raw values do not appear in the result; the matrix contains their standardized pairwise relationships.
Common mistake avoided: dividing centered values by their variances rather than their standard deviations. For \(X_1\), that mistake would give a standardized column with sample variance \(1/(400/3)=0.0075\), not one. The correct divisor is \(s_1=20/\sqrt{3}\).
Calculating Component Scores
Once the eigenvectors of the correlation matrix are known, each observation must first be represented in standardized coordinates. The score is a weighted sum of those standardized values. A useful check is to multiply the resulting component scores by their eigenvectors: using all components should recover the original standardized observation, apart from rounding.
Worked Example: Scoring a Standardized Observation
Use the directions from the earlier three-variable matrix: \[ v_1=(1,1,0)^{\mathsf T}/\sqrt{2},\quad v_2=(0,0,1)^{\mathsf T},\quad v_3=(1,-1,0)^{\mathsf T}/\sqrt{2}. \] A new hypothetical observation has standardized values \(z=(2,0,1)\). Calculate its three component scores and check the result.
First component. The score is \(zv_1=(2+0)/\sqrt{2}=\sqrt{2}\).
Second component. The score is \(zv_2=1\).
Third component. The score is \(zv_3=(2-0)/\sqrt{2}=\sqrt{2}\).
The scores are therefore \((\sqrt{2},1,\sqrt{2})\). To check, reconstruct the standardized vector: \[ \sqrt{2}v_1+1v_2+\sqrt{2}v_3 =(1,1,0)+(0,0,1)+(1,-1,0)=(2,0,1). \] This equals the original standardized observation. The calculation also confirms that component scores are projections of standardized data onto the eigenvector directions.
Common mistake avoided: treating an eigenvector entry as a score. The entries of \(v_k\) define the direction; the score depends on both that direction and the particular observation.
Interpretation and Checks
Correlation-based PCA describes patterns in the standardized variables. A coefficient in an eigenvector weights one standardized variable in a component direction; it is not a weight applied directly to that variable in its original units. A positive association between two variables may lead a direction to give them coefficients with the same sign, while a contrast between them may give opposite signs.
Before accepting a calculation, check that the matrix is symmetric, has ones on its diagonal, and has correlations within the interval \([-1,1]\). Check that the eigenvectors have unit length and are mutually orthogonal. Finally, verify \(Rv_k=\lambda_kv_k\) for the reported eigenpairs. These checks can reveal transcription errors or a score calculation performed with the wrong coordinates.
Correlation-based PCA depends on the correlations among the variables, not on their original measurement scales. It does not eliminate dependence, guarantee that a component has a particular practical interpretation, or make a constant variable usable: if a variable has zero sample standard deviation, it cannot be standardized by this method. These distinctions matter whenever results are calculated or interpreted from a correlation matrix.
Check Your Understanding
Work through each problem, paying attention to whether the supplied values are raw or already standardized.
- A two-variable correlation matrix has off-diagonal entries \(0.4\). What are its eigenvalues, and what unit direction corresponds to the larger eigenvalue? (Answer: eigenvalues \(1.4\) and \(0.6\); the larger-eigenvalue direction is \((1,1)^{\mathsf T}/\sqrt{2}\).)
- For \(n\) complete observations, each column of \(Z\) is centered and divided by its sample standard deviation. What matrix equals \(Z^{\mathsf T}Z/(n-1)\), and what condition is required of each column before standardizing? (Answer: the sample correlation matrix \(R\); every included variable must have positive sample standard deviation.)
- For \(R=\begin{pmatrix}1&0&0\\0&1&0.2\\0&0.2&1\end{pmatrix}\), give the eigenvalues in descending order and a unit direction for each. (Answer: \(1.2\) with \((0,1,1)^{\mathsf T}/\sqrt{2}\); \(1\) with \((1,0,0)^{\mathsf T}\); \(0.8\) with \((0,1,-1)^{\mathsf T}/\sqrt{2}\).)
- Two centered columns have cross-product \(12\), and their sums of squares are \(16\) and \(25\). Calculate their sample correlation. (Answer: \(12/\sqrt{16\cdot25}=12/20=0.6\).)
- An observation has standardized values \((1,2)\), and a component direction is \((1,-1)^{\mathsf T}/\sqrt{2}\). Calculate its score. (Answer: \((1-2)/\sqrt{2}=-1/\sqrt{2}\).)