Why Strong Correlation Changes the PCA Picture
When several measurements tend to rise and fall together, much of their variation may be summarized by a direction that gives them weights of the same sign. In principal components analysis (PCA), this direction can form the first principal component: a summary of their shared movement. Later components are constrained to be orthogonal to the first, so they often describe differences or contrasts among the measurements instead.
Consider, for example, hypothetical standardized measurements of several aspects of a device’s operating load. If all measurements are positively correlated, an observation that is high on one will tend to be high on the others. A component that adds the measurements can capture this common pattern. It does not follow that every strongly correlated data set has exactly the same component structure: the actual eigenvectors and eigenvalues depend on the correlation matrix.
If \(R\) has eigenvalues \(\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\), then the proportion of total standardized variance explained by component \(j\) is \(\lambda_j/\sum_{k=1}^{p}\lambda_k\). For a correlation matrix, the diagonal entries are one, so \(\sum_{k=1}^{p}\lambda_k=\operatorname{tr}(R)=p\). This tutorial uses correlation matrices or already-standardized measurements; the interpretation would require care if variables were left on scales that make their variances incomparable.
Why an Eigenvector Gives the Maximum-Variance Direction
The variance of the score \(w^\mathsf{T}z\) is \(w^\mathsf{T}Rw\), because the covariance matrix of the standardized vector \(z\) is \(R\). To find the first component, maximize this variance while requiring the direction to have unit length. The length constraint matters: without it, multiplying \(w\) by a large constant would make the score variance arbitrarily large.
Maximize \(w^\mathsf{T}Rw\) subject to \(w^\mathsf{T}w=1\).
Using a Lagrange multiplier \(\lambda\), set the derivative of \(w^\mathsf{T}Rw-\lambda(w^\mathsf{T}w-1)\) equal to zero. Since \(R\) is symmetric, this gives \(2Rw-2\lambda w=0\), or \(Rw=\lambda w\).
The stationary directions are eigenvectors of \(R\). For a unit eigenvector \(w\) with eigenvalue \(\lambda\), \(w^\mathsf{T}Rw=w^\mathsf{T}(\lambda w)=\lambda\). Thus, the largest eigenvalue gives the maximum score variance.
The same reasoning, with orthogonality constraints for earlier directions, produces the later components. This explains an important interpretation: a component is a weighted combination of measurements, not a newly observed original variable.
Two Strongly Correlated Measurements
For two standardized measurements with correlation \(r\), their correlation matrix is \(R=\begin{pmatrix}1&r\\r&1\end{pmatrix}\). When \(r\) is positive and large, the direction that adds the two measurements has the larger eigenvalue. The orthogonal direction compares them. The following example works out the exact result.
Worked Example: Shared Movement in Two Standardized Measurements
A hypothetical equipment study records two standardized operating measurements, \(Z_1\) and \(Z_2\). Their correlation matrix is \(R=\begin{pmatrix}1&0.9\\0.9&1\end{pmatrix}\). Find both principal component directions and eigenvalues, the variance explained by the first component, and the meaning of the two directions. The measurements are already standardized, and the supplied matrix is the correlation matrix to analyze.
Method. Solve the eigenvalue equation \(\det(R-\lambda I)=0\). Here, \[ \det(R-\lambda I) =(1-\lambda)^2-0.9^2 =1.81-2\lambda+\lambda^2. \] Setting this equal to zero gives \((1.9-\lambda)(0.1-\lambda)=0\). The eigenvalues are therefore \(\lambda_1=1.9\) and \(\lambda_2=0.1\).
For \(\lambda_1=1.9\), the first row of \(R-\lambda_1 I\) is \((-0.9,0.9)\), so an eigenvector has equal coordinates. Normalizing it to length one gives \(w_1=(1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\). For \(\lambda_2=0.1\), the first row is \((0.9,0.9)\), so the coordinates must be opposites. A unit eigenvector is \(w_2=(1/\sqrt{2},-1/\sqrt{2})^\mathsf{T}\).
The total variance is \(1+1=2\), also equal to \(1.9+0.1\). Thus, the first component explains \(1.9/2=0.95\), or 95% of the total standardized variance; the second explains \(0.1/2=0.05\), or 5%.
The first score is \((Z_1+Z_2)/\sqrt{2}\). Its equal positive weights describe shared movement: observations high on both measurements tend to have high first-component scores. The second score is \((Z_1-Z_2)/\sqrt{2}\), a contrast that is high when one measurement exceeds the other. The eigenvalues sum to 2, as required by the trace of this two-variable correlation matrix. The common mistake to avoid is treating the 0.9 correlation itself as the proportion explained; the first component explains 0.95, calculated from its eigenvalue divided by total variance.
Scores and One-Component Reconstruction
An eigenvector describes a direction; a component score describes the position of an individual observation along that direction. For a standardized observation \(z\), the score on a unit direction \(w_j\) is \(t_j=w_j^\mathsf{T}z\). Retaining only the first component approximates the observation by \(t_1w_1\). The difference \(z-t_1w_1\) is the part not represented by that one-component reconstruction.
Worked Example: Separating Common Level From a Contrast
Using the hypothetical two-measurement study and its matrix from the previous calculation, consider a particular standardized observation \(z=(1.2,0.8)^\mathsf{T}\). Calculate its scores on both components, reconstruct it using only the first component, and check the discarded part.
The directions are \(w_1=(1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\) and \(w_2=(1/\sqrt{2},-1/\sqrt{2})^\mathsf{T}\). The first score is \(t_1=w_1^\mathsf{T}z=(1.2+0.8)/\sqrt{2}=2/\sqrt{2}=\sqrt{2}\). The second score is \(t_2=w_2^\mathsf{T}z=(1.2-0.8)/\sqrt{2}=0.4/\sqrt{2}=0.2\sqrt{2}\).
The first-component reconstruction is \(t_1w_1=(\sqrt{2}/\sqrt{2},\sqrt{2}/\sqrt{2})^\mathsf{T}=(1,1)^\mathsf{T}\). Subtracting it from the original observation gives the residual \((1.2,0.8)^\mathsf{T}-(1,1)^\mathsf{T}=(0.2,-0.2)^\mathsf{T}\). Its squared length is \(0.2^2+(-0.2)^2=0.08\).
The second component accounts for this difference: \(t_2w_2=(0.2,-0.2)^\mathsf{T}\). Therefore, \(z=t_1w_1+t_2w_2\), and the one-component reconstruction error is \(0.08=t_2^2\). The first score reflects the observation’s overall positive level, while the smaller second score records that \(Z_1\) is somewhat higher than \(Z_2\). The method is projection onto a unit eigenvector; the trap is to use the score alone as the reconstructed observation and omit multiplication by its direction.
More Than Two Measurements: A Shared Component and Contrast Directions
With three or more measurements, the same idea applies, but there can be several orthogonal directions for contrasts. In a particularly clear special case, all distinct pairs of standardized measurements have the same positive correlation \(\rho\). The correlation matrix then has ones on its diagonal and \(\rho\) everywhere else. The all-equal direction is an eigenvector with eigenvalue \(1+(p-1)\rho\); every direction whose coordinates sum to zero is a contrast direction with eigenvalue \(1-\rho\). Thus, when \(\rho\) is large and positive, the first direction receives much more variance than any one contrast direction. This special structure is useful for interpretation, but real correlation matrices need not have equal correlations.
Worked Example: Three Measurements With a Common Positive Correlation
A hypothetical monitoring system records three standardized indicators of operating demand. Each pair has correlation 0.8, so the correlation matrix is \(R=\begin{pmatrix}1&0.8&0.8\\0.8&1&0.8\\0.8&0.8&1\end{pmatrix}\). Find the first component’s direction and explained variance, identify the contrast variance, and reconstruct the standardized observation \(z=(1.5,1.0,0.5)^\mathsf{T}\) from its first component alone.
For the vector of ones, each entry of \(R(1,1,1)^\mathsf{T}\) is \(1+0.8+0.8=2.6\). Therefore, the unit first direction is \(w_1=(1,1,1)^\mathsf{T}/\sqrt{3}\), with eigenvalue 2.6. The total variance is the trace, \(3\), so the first component explains \(2.6/3=0.8667\), or 86.67% when expressed as a percentage to two decimals.
For a contrast direction \(w\) whose coordinates sum to zero, write \(R=0.2I+0.8J\), where \(I\) is the identity matrix and \(J\) is the all-ones matrix. Since \(Jw=0\), \(Rw=0.2w\). The two independent contrast directions therefore have eigenvalue 0.2 each. Together they explain \(0.4/3=0.1333\), or 13.33% of total variance. The two contrast eigenvalues are equal, so there is no unique choice of individual orthogonal directions within their two-dimensional contrast space.
The first score for the stated observation is \(t_1=w_1^\mathsf{T}z=(1.5+1.0+0.5)/\sqrt{3}=3/\sqrt{3}=\sqrt{3}\). The one-component reconstruction is \(t_1w_1=(\sqrt{3}/\sqrt{3})(1,1,1)^\mathsf{T}=(1,1,1)^\mathsf{T}\). The residual is \((0.5,0,-0.5)^\mathsf{T}\), with squared length \(0.5^2+0^2+(-0.5)^2=0.5\). The two contrast eigenvalues sum to 0.4, but the squared residual for this particular observation is 0.5; these quantities answer different questions. The former summarizes variance across the data, while the latter measures reconstruction error for this observation. The eigenvalues sum to \(2.6+0.2+0.2=3\), which checks against the trace.
What Strong Correlation Does—and Does Not—Guarantee
A large positive correlation often produces a first component with loadings of the same sign, but the details depend on the full matrix. The two-variable case gives an exact result: with \(r>0\), the equal-weight direction is first, and as \(r\) approaches one, its eigenvalue approaches two while the contrast eigenvalue approaches zero. With more variables, unequal correlations can produce unequal loadings; the appropriate direction must be found from the actual eigenvectors rather than assumed to assign equal weights.
A large first explained-variance ratio is evidence that one direction summarizes much of the variation in the standardized measurements. It is not proof that the component is a causal factor, a directly observed quantity, or the only useful summary. A component score is a mathematical combination of the input measurements. Its substantive interpretation depends on the measurement context and the loading pattern.
Also distinguish variance explained across observations from the reconstruction error for one observation. Eigenvalues describe component variances for the data being analyzed. The residual after projecting a particular observation onto the first component describes that observation’s distance from the retained direction. The calculations in the examples show why both quantities are useful, and why they should not be substituted for one another.
Check Your Understanding
For each problem, use the stated standardized measurements and correlation matrix. Check that eigenvalues sum to the matrix trace.
- Two standardized inspection readings have correlation 0.8. Give the two unit component directions, their eigenvalues, and the first component’s proportion of total variance. (Answer: directions \((1,1)^\mathsf{T}/\sqrt{2}\) and \((1,-1)^\mathsf{T}/\sqrt{2}\); eigenvalues 1.8 and 0.2; first proportion \(1.8/2=0.9000\).)
- For the two-reading matrix with correlation 0.8, an observation is \(z=(1.4,0.6)^\mathsf{T}\). Find its first-component score and one-component reconstruction. (Answer: score \(2/\sqrt{2}=\sqrt{2}\); reconstruction \((1,1)^\mathsf{T}\).)
- Three standardized sensor readings have every pairwise correlation equal to 0.7. Find the eigenvalue for the all-equal direction, each contrast eigenvalue, and the first explained-variance proportion. (Answer: eigenvalues \(2.4,0.3,0.3\); first proportion \(2.4/3=0.8000\).)
- For the three-reading matrix in question 3, project \(z=(1.2,0.9,0.9)^\mathsf{T}\) onto the all-equal first direction. Give the score, reconstruction, and squared reconstruction error. (Answer: score \(3/\sqrt{3}=\sqrt{3}\); reconstruction \((1,1,1)^\mathsf{T}\); squared error \(0.2^2+(-0.1)^2+(-0.1)^2=0.0600\).)
- For three standardized measurements with common pairwise correlation 0.6, a candidate reports first eigenvalue 2.2 and two contrast eigenvalues 0.4 each. Is the result consistent with the matrix, and what proportion of total variance is explained by the first component? (Answer: Yes; the eigenvalues sum to 3 and the first proportion is \(2.2/3=0.7333\).)