PCA of Highly Correlated Variables

Why Strong Correlation Changes the PCA Picture

When several measurements tend to move together, each may repeat much of the same information. Principal components analysis (PCA) expresses the variation in those measurements using new, uncorrelated directions. For highly correlated measurements, the leading direction often represents their shared movement, while later directions describe differences among them.

The word “often” matters. High pairwise correlations do not, by themselves, guarantee that the first component has equal weights or that every later component is unimportant. The exact relationships depend on the full covariance or correlation matrix. The example here uses a deliberately symmetric correlation structure so the component directions and their meanings can be derived directly.

Definition: For a data matrix with \(p\) numeric variables, PCA of the centered variables finds orthonormal directions \(v_1,\ldots,v_p\) that successively maximize the variance of projected observations, subject to being orthogonal to previously chosen directions. The component score for an observation \(z\) on direction \(v_j\) is \(t_j=v_j^\mathsf{T}z\). For correlation-based PCA, the variables are standardized first, and the directions are eigenvectors of their correlation matrix.

Here, \(z\) denotes an observation expressed in standardized coordinates. A standardized value of 1 means one standard deviation above the variable’s mean; 0 means the mean. As established in the earlier material on centering and scaling, the choice to standardize is consequential: correlation-based PCA gives variables equal variance before examining their joint pattern. It is appropriate when the goal is to compare their correlation structure rather than let their original measurement units determine the result.

PCA is a descriptive linear transformation, not a probability model requiring normally distributed observations. The calculations require numeric variables with finite variances. In practice, the estimated correlations should be based on comparable records and interpreted in light of how the measurements were collected. PCA summarizes linear co-movement; it does not establish that one measurement causes another.

A Symmetric Example of Strong Correlation

Suppose three standardized measurements \(Z_1,Z_2,Z_3\) have pairwise correlation \(0.8\). For example, they might be three hypothetical measurements of related aspects of the same process. Their correlation matrix is

R= \begin{pmatrix} 1 & 0.8 & 0.8\\ 0.8 & 1 & 0.8\\ 0.8 & 0.8 & 1 \end{pmatrix}.

The diagonal entries equal 1 because each standardized variable has variance 1. The off-diagonal entries say that every pair has correlation 0.8. This is a strong, positive, and especially regular pattern: all variables correlate equally with one another. Do not assume this symmetry for real data merely because all correlations are positive.

Worked Example: Finding the Shared Direction

For the stated three-variable correlation matrix, find the first principal component’s direction, its variance, and the proportion of total variance it represents. Then find the remaining eigenvalues and their variance proportions.

Method. Find the eigenvalues and orthonormal eigenvectors of \(R\). For a correlation-based PCA, each eigenvalue is the variance of scores in its component direction, and the total variance is the trace of \(R\), here \(1+1+1=3\).

Let \(v_1=(1,1,1)^\mathsf{T}/\sqrt{3}\). Each row of \(R\) sums to \(1+0.8+0.8=2.6\), so multiplying \(R\) by the all-ones vector multiplies it by 2.6. Therefore \(v_1\) is an eigenvector with eigenvalue \(\lambda_1=2.6\). Its entries have equal size and the same sign: this component is the common movement of the three standardized measurements.

Any vector whose entries sum to zero is orthogonal to \((1,1,1)^\mathsf{T}\). For such a vector \(w\), the sum of the entries contributed to each row by the off-diagonal terms is \(-0.8w_i\), because the other two entries sum to \(-w_i\). Including the diagonal term gives \(Rw=(1-0.8)w=0.2w\). Thus both independent directions orthogonal to \(v_1\) have eigenvalue \(0.2\).

The total variance is 3. The first component’s proportion is \(2.6/3=0.8667\), or 86.67% to two decimal places. Each later component’s proportion is \(0.2/3=0.0667\), or 6.67%. The eigenvalues sum to \(2.6+0.2+0.2=3\), as required.

Check and trap avoided. The leading eigenvalue is not 2.6 divided by the number of variables; 2.6 is already the component variance. Divide it by the total variance, 3, only to obtain its proportion. The small equal eigenvalues describe two contrast directions; they do not mean those directions have exactly zero variation.

Why the Leading Direction Maximizes Variance

The defining optimization gives a short justification for why eigenvectors appear. Let \(Z\) be a centered random vector with covariance matrix \(\Sigma\), and consider any unit direction \(v\), so \(v^\mathsf{T}v=1\). The projected value is \(v^\mathsf{T}Z\). Its variance is

\operatorname{Var}(v^\mathsf{T}Z) =E[(v^\mathsf{T}Z)^2] =v^\mathsf{T}\Sigma v.

The first equality uses the zero mean of \(Z\); the second follows by expanding the scalar product and applying the definition of the covariance matrix. By the spectral decomposition of a symmetric covariance matrix, write \(v=\sum_j a_jv_j\), where \(v_j\) are orthonormal eigenvectors and \(\sum_j a_j^2=1\). Then

v^\mathsf{T}\Sigma v =\sum_j\lambda_j a_j^2 \leq \lambda_1\sum_j a_j^2 =\lambda_1,

where \(\lambda_1\) is the largest eigenvalue. Equality is attained by choosing \(v=v_1\). This proves that the leading eigenvector gives the direction of greatest variance. Requiring the next direction to be orthogonal to \(v_1\) gives the same argument on the remaining eigenspace, and selects a direction with the next-largest eigenvalue.

Component Scores: A Record’s Shared and Contrasting Pattern

For the symmetric matrix above, one convenient orthonormal set of directions is

v_1=\frac{(1,1,1)^\mathsf{T}}{\sqrt{3}}, \qquad v_2=\frac{(1,-1,0)^\mathsf{T}}{\sqrt{2}}, \qquad v_3=\frac{(1,1,-2)^\mathsf{T}}{\sqrt{6}}.

The first direction measures the shared level. The second contrasts the first measurement with the second. The third contrasts the average of the first two with the third. These descriptions follow from the directions’ coefficients, not from any claim that one measurement causes another. Since the last two eigenvalues are equal, different orthonormal choices within their two-dimensional contrast space are also valid; the individual contrast directions are not uniquely determined by this matrix.

Worked Example: Calculating Scores and Checking the Transformation

A hypothetical record has standardized values \(z=(1,0.8,1.2)^\mathsf{T}\). Calculate its three scores using the directions above. Check that the squared scores sum to the squared length of the original standardized record.

Method. Take the dot product of \(z\) with each unit direction. This projects the record onto an orthonormal basis, so its squared length must be preserved when all three component scores are included.

For the shared component, \(t_1=v_1^\mathsf{T}z=(1+0.8+1.2)/\sqrt{3}=3/\sqrt{3}=\sqrt{3}\), approximately 1.7321.

For the first contrast, \(t_2=v_2^\mathsf{T}z=(1-0.8)/\sqrt{2}=0.2/\sqrt{2}\), approximately 0.1414.

For the second contrast, \(t_3=v_3^\mathsf{T}z=(1+0.8-2(1.2))/\sqrt{6}=-0.6/\sqrt{6}\), approximately -0.2449. A negative score means the record lies on the negative side of the selected direction; reversing an eigenvector’s sign would reverse the score without changing the PCA.

Use the exact scores for the check: \(t_1^2+t_2^2+t_3^2=3+(0.2^2/2)+(0.6^2/6)=3+0.02+0.06=3.08\). The original squared length is \(1^2+0.8^2+1.2^2=1+0.64+1.44=3.08\), so the check agrees. The displayed decimal scores are rounded approximations; squaring those rounded values is not the exact check.

Check and trap avoided. The dot product for \(t_3\) includes the coefficient \(-2\) on the third measurement. Omitting it would change both the score and the interpretation of this contrast. Also, the component score is not the eigenvalue: the eigenvalue describes variance across observations, while \(t_j\) describes this particular record’s coordinate.

What the Later Components Still Retain

For the matrix in this example, the first component carries most of the total standardized variance, but the two later components account for \(0.2+0.2=0.4\) of the total 3, or 13.33%. Their scores record deviations from perfectly shared movement. A record can therefore be close to the overall shared pattern and still have a meaningful contrast among its measurements.

The reconstruction from only the first component is \(t_1v_1\). The omitted components together give the residual \(t_2v_2+t_3v_3\). This makes precise what is lost by representing a record with the shared direction alone: the difference between its measurements that cannot be expressed as an equal shift in all three.

Worked Example: Quantifying the Information in the Contrasts

Use the record \(z=(1,0.8,1.2)^\mathsf{T}\) and its scores from the preceding example. Reconstruct it using only the first component, calculate the omitted squared error, and explain what the omitted part represents.

Method. Project onto \(v_1\) by multiplying \(t_1v_1\). Because the directions are orthonormal, the squared reconstruction error from omitting components 2 and 3 is \(t_2^2+t_3^2\).

Since \(t_1=\sqrt{3}\) and \(v_1=(1,1,1)^\mathsf{T}/\sqrt{3}\), the one-component reconstruction is \(t_1v_1=(1,1,1)^\mathsf{T}\). Subtracting it from the observed record gives residual \((0,-0.2,0.2)^\mathsf{T}\). Its squared length is \(0^2+(-0.2)^2+0.2^2=0.08\).

The score calculation gives the same result: \(t_2^2+t_3^2=0.2^2/2+0.6^2/6=0.02+0.06=0.08\). The first-component reconstruction has squared error 0.08 for this record, while the full record has squared length 3.08. The omitted part expresses that the second measurement is 0.2 below the shared level and the third is 0.2 above it.

Check and trap avoided. The 13.33% figure is the proportion of variance across the full data set in both later directions, not the error percentage for every individual record. This record’s squared error relative to its own squared length is \(0.08/3.08\), approximately 2.60%. The two percentages answer different questions.

Interpretation and Exam Traps

A leading component with same-sign coefficients is consistent with a shared increase or decrease in the standardized measurements. It is not proof that the measurements represent one underlying cause. The component is a mathematical summary of their observed linear pattern. In this example the equal coefficients arise from the deliberately equal pairwise correlations; with unequal correlations, the leading direction need not weight variables equally.

The later components in a highly correlated set commonly describe contrasts, but “later” does not mean “irrelevant.” A small amount of total variance can carry a distinction important to the question being asked. PCA alone cannot decide whether that distinction matters for a particular application. It is also incorrect to treat a low-variance component as a component with no information: its scores still describe observed variation, although they explain less of the overall spread.

Confusing an eigenvalue with a percentage. Divide an eigenvalue by the sum of all eigenvalues to get its variance proportion.
Calling every first component a shared level. That interpretation is warranted by the same-sign, similar weights in this example, not by the label “PC1” alone.
Discarding contrasts automatically. Later components contain differences among variables; whether those differences can be set aside depends on the purpose of the analysis.
Ignoring standardization. The example is based on a correlation matrix, so every variable has unit variance. PCA on an unscaled covariance matrix can give a different result when measurement scales differ.
Overinterpreting signs. The sign of an eigenvector may be reversed along with all its scores, leaving the component unchanged. Interpret the pattern of relative signs and magnitudes, not an absolute positive or negative orientation.

Key takeaway: With strongly and symmetrically correlated standardized variables, the leading component can capture their common movement, while later components retain contrasts. Read the full eigenstructure and the score patterns: a high leading variance proportion does not imply that every omitted difference is unimportant.

Check Your Understanding

Work each problem from the stated matrix or values. Answers are given at the end of each item.

Three standardized variables have equal pairwise correlation 0.7. Find the eigenvalue for the shared direction and its proportion of total variance. (Answer: eigenvalue 2.4; proportion \(2.4/3=0.8000\), or 80.00%.)
For the three directions \(v_1,v_2,v_3\) defined above, a standardized record is \(z=(0.9,0.7,1.1)^\mathsf{T}\). Find its three scores, giving each to four decimals. (Answer: \(t_1=2.7/\sqrt{3}=1.5588\), \(t_2=0.2/\sqrt{2}=0.1414\), \(t_3=-0.6/\sqrt{6}=-0.2449\).)
Two standardized variables have correlation 0.6. What are the eigenvalues of their correlation matrix, and what proportion of total variance belongs to the contrast direction? (Answer: eigenvalues 1.6 and 0.4; the contrast proportion is \(0.4/2=0.2000\), or 20.00%.)
For \(z=(1.1,0.9,0.5)^\mathsf{T}\), reconstruct using only \(v_1\), then find the squared reconstruction error. (Answer: the reconstruction is \((2.5/3,2.5/3,2.5/3)^\mathsf{T}\); squared error is \(0.02+1/6=0.1867\), to four decimals.)
In the three-variable equal-correlation example, why should a small score on a later component not be interpreted as evidence that the corresponding measurement is unimportant? (Answer: a score is record-specific, and the later directions represent contrasts; the overall eigenvalue measures their variance across records, not their application-specific importance.)

What You'll Learn