PCA of Weakly Correlated Variables

When PCA Finds Little Concentrated Structure

PCA is most visibly useful when a few components capture a large share of the variation in many variables. If the variables are only weakly correlated, their variation may instead be spread across several components. In that case, the first component can still be calculated, but its loadings may be diffuse and its share of variance modest. The practical question is not simply whether PCA produces components; it is whether the component representation gives a useful reduction in dimension.

This tutorial considers PCA of standardized variables, so the relevant matrix is their correlation matrix. That choice puts the variables on a common scale. As in the earlier material on interpreting loadings, a loading describes a component direction, not a causal effect or proof that the variables measure one underlying construct.

Definition: Let \(Z_1,\ldots,Z_p\) be centered, standardized variables with correlation matrix \(R\). A principal component direction is a unit eigenvector \(v_k\) of \(R\), with eigenvalue \(\lambda_k\), where \(Rv_k=\lambda_kv_k\). The component score is \(v_k^\mathsf{T}Z\), and its variance is \(\lambda_k\). Since the diagonal entries of a correlation matrix are 1, the total variance is \(\operatorname{tr}(R)=p\). Thus component \(k\)'s explained-variance proportion is \(\lambda_k/p\).

Eigenvalues are conventionally ordered from largest to smallest. A component's loadings are the entries of its unit eigenvector; their squared entries sum to 1. When the entries have similar magnitudes, the direction is diffuse: it involves many variables rather than being dominated by just one or two. Diffuse loadings do not, by themselves, imply that a substantial amount of total variance is concentrated in that direction.

Why Weak Correlations Often Mean Modest Variance Shares

A useful exact case is a correlation matrix with the same pairwise correlation \(r\) between every distinct pair of variables:

R=(1-r)I_p+r\mathbf{1}\mathbf{1}^{\mathsf{T}},

Here \(I_p\) is the \(p\)-dimensional identity matrix and \(\mathbf{1}\) is the vector of \(p\) ones. Multiplying by \(\mathbf{1}\) gives

R\mathbf{1} =(1-r)\mathbf{1}+r\mathbf{1}(\mathbf{1}^{\mathsf{T}}\mathbf{1}) =\bigl(1+(p-1)r\bigr)\mathbf{1}.

So \(\mathbf{1}\) is an eigenvector with eigenvalue \(1+(p-1)r\). For any vector \(w\) perpendicular to \(\mathbf{1}\), \(\mathbf{1}^{\mathsf{T}}w=0\), and therefore

Rw=(1-r)w+r\mathbf{1}(\mathbf{1}^{\mathsf{T}}w)=(1-r)w.

There are \(p-1\) independent directions perpendicular to \(\mathbf{1}\), each with eigenvalue \(1-r\). This derivation also shows why the sign of \(r\) matters: the eigenvalue along \(\mathbf{1}\) is the largest when \(r>0\); when \(r=0\), all eigenvalues equal 1; and when \(r<0\), the perpendicular directions have the larger eigenvalue. In particular, one must not automatically call \(\mathbf{1}\) a first-component direction.

Worked Example: Equal, Weakly Positive Correlations

Suppose three standardized measurements have the correlation matrix

R= \begin{pmatrix} 1&0.10&0.10\\ 0.10&1&0.10\\ 0.10&0.10&1 \end{pmatrix}.

Method. This is an equal-correlation matrix with \(p=3\) and \(r=0.10\). The eigenvalue in the \(\mathbf{1}\) direction is \(1+(p-1)r\), so it is \(1+2(0.10)=1.20\). The other two eigenvalues are \(1-r=0.90\). The eigenvalues sum to \(1.20+0.90+0.90=3\), which agrees with the trace of \(R\).

The first component's unit direction is \((1,1,1)^\mathsf{T}/\sqrt{3}\), so it has three equal, positive loadings. Its variance proportion is \(1.20/3=0.40\), or 40%. Each of the other components accounts for \(0.90/3=0.30\), or 30%. Keeping only the first component therefore leaves 60% of the total standardized variance unrepresented.

Check. The proportions sum to \(0.40+0.30+0.30=1\), and the first eigenvalue is the largest because \(r>0\). The equal loadings describe a shared direction, but the 40% share shows that this direction does not summarize most of the total variation. The common mistake is to infer that equal loadings make the component a strong summary; the variance proportion, not the appearance of the loading vector, measures how much variation it captures.

For fixed \(p\), a small positive \(r\) makes the first component's share \(\{1+(p-1)r\}/p\), which may be only modestly greater than the \(1/p\) share each component would have if the variables were uncorrelated. With many variables, even small positive correlations can raise the first eigenvalue noticeably above 1; that does not necessarily make its share of the total variance large. The eigenvalue must be compared with the total \(p\).

Worked Example: Weak Correlations in Two Separate Pairs

A hypothetical analysis has four standardized variables. Variables 1 and 2 have correlation \(0.12\), variables 3 and 4 have correlation \(0.08\), and correlations between the two pairs are zero. Thus

R= \begin{pmatrix} 1&0.12&0&0\\ 0.12&1&0&0\\ 0&0&1&0.08\\ 0&0&0.08&1 \end{pmatrix}.

Method. For a two-variable block with correlation \(c\), the equal-sign direction has eigenvalue \(1+c\) and the opposite-sign direction has eigenvalue \(1-c\). The first block therefore contributes eigenvalues \(1.12\) and \(0.88\); the second contributes \(1.08\) and \(0.92\). In descending order, the four eigenvalues are \(1.12,1.08,0.92,0.88\). Their sum is 4, as required.

The explained-variance proportions are \(1.12/4=28\%\), \(1.08/4=27\%\), \(0.92/4=23\%\), and \(0.88/4=22\%\). The cumulative proportions are 28%, 55%, 78%, and 100%. The first component describes the shared direction of variables 1 and 2; the second describes the shared direction of variables 3 and 4. Neither one dominates the total variation.

Check. Retaining three components keeps 78% of the variance but reduces four dimensions to three. The calculation is consistent with the eigenvalues summing to the total variance. A common mistake is to treat the two pairwise patterns as though one component captures both: the block structure gives them separate directions, with eigenvalues 1.12 and 1.08.

Reading Diffuse Loadings and Retention Together

A diffuse first-component loading pattern is common when several variables contribute similarly to a direction. Its interpretation should be paired with the eigenvalue and the number of variables. For example, if \(p=5\), an eigenvalue of 1.25 represents \(1.25/5=25\%\) of the total variance. It exceeds the average eigenvalue of 1, but still leaves 75% outside that component. An eigenvalue above 1 is not, on its own, evidence that a low-dimensional summary is adequate.

Retention decisions should also account for how much compression is achieved. Retaining four of five components may preserve a large cumulative share, but it discards only one dimension. Conversely, retaining one or two components may achieve a substantial reduction in dimension but lose a large share when the eigenvalues are similar. There is no universal cutoff that makes a weakly correlated data set suitable for reduction: the retained variance, number of components, and clarity of their loading patterns all matter.

Worked Example: Does a Four-Component Summary Reduce Five Dimensions?

Suppose a hypothetical PCA of five standardized measurements reports eigenvalues \(1.32,1.18,0.96,0.82,0.72\). The leading loading vector, rounded to two decimals, is \((0.50,0.48,0.46,0.43,0.35)^\mathsf{T}\). Assess what can reasonably be concluded from these results.

Method. First check the total: \(1.32+1.18+0.96+0.82+0.72=5.00\), as expected for a five-variable correlation matrix. The component proportions are \(1.32/5=26.4\%\), \(1.18/5=23.6\%\), \(0.96/5=19.2\%\), \(0.82/5=16.4\%\), and \(0.72/5=14.4\%\). Their cumulative proportions are 26.4%, 50.0%, 69.2%, 85.6%, and 100.0%.

The first loading vector is diffuse: every variable has a positive loading, and none dominates. Its entries are rounded, so their squared sum is approximately \(0.50^2+0.48^2+0.46^2+0.43^2+0.35^2=0.9994\), close to 1 as expected for a unit eigenvector. The modest 26.4% variance share means that this broad direction does not summarize most of the total variation.

If the aim is to retain at least 80% of the variance, four components are required: three retain 69.2%, while four retain 85.6%. This reduces five dimensions to four, so it may not be a compelling reduction even though the cumulative share exceeds 80%.

Check. The component proportions sum to 100%, and the stated four-component total is \(26.4+23.6+19.2+16.4=85.6\%\). A common mistake is to divide the leading eigenvalue by the second eigenvalue, obtaining \(1.32/1.18\), and call that a variance proportion. The correction is to divide by the sum of all eigenvalues, 5; the leading component explains \(1.32/5=26.4\%\), not a proportion greater than 100%.

Exam Traps and Practical Interpretation

When correlations are weak, check the whole eigenvalue spectrum rather than judging the PCA from its first direction alone. Similar eigenvalues indicate that variance is distributed across components. A leading eigenvalue slightly above 1 can still correspond to a small share of total variance, especially when \(p\) is not small. Diffuse loadings may be easy to describe, but they do not establish that the data have a strong low-dimensional structure.

The equal-correlation case also provides a useful sign check. For \(p=3\) and \(r=-0.10\), the eigenvalue along \(\mathbf{1}\) is \(1+2(-0.10)=0.80\), while each perpendicular direction has eigenvalue \(1-(-0.10)=1.10\). The first component therefore lies in the perpendicular eigenspace, not along \(\mathbf{1}\). At \(r=0\), all three eigenvalues equal 1, so there is no uniquely determined first direction. This is a frequent trap when applying the equal-correlation formula without considering the sign of \(r\).

A sound conclusion distinguishes the mathematical result from the practical decision: identify the leading directions and their variance shares, then state how much dimension reduction a proposed retention choice actually achieves. If most components are needed to retain the desired share, the data may not support a useful low-dimensional summary. PCA still provides a valid rotation of the variables, but a rotation alone is not a reason to discard dimensions.

Key takeaway: With weak correlations, variance is often spread across several components. Interpret loadings alongside eigenvalues and cumulative variance, verify that the eigenvalue denominator is the total \(p\), and assess both retained variance and the amount of dimension reduction.

Check Your Understanding

For each problem, calculate the eigenvalues or variance shares requested and decide what the results say about dimension reduction.

Three standardized variables have a common pairwise correlation of \(0.05\). Find the eigenvalues and the first component's explained-variance proportion. (Answer: eigenvalues \(1.10, 0.95, 0.95\); first component \(36.67\%\).)
Four standardized variables have a common pairwise correlation of \(-0.05\). Which eigenspace contains the first component, and what proportion of variance does it explain? (Answer: the three-dimensional space perpendicular to \(\mathbf{1}\); eigenvalue \(1.05\), explained variance \(26.25\%\).)
A four-variable correlation matrix consists of two independent pairs, with within-pair correlations \(0.16\) and \(0.08\). Find the ordered eigenvalues and the cumulative variance after two components. (Answer: eigenvalues \(1.16,1.08,0.92,0.84\); two components retain \(56\%\).)
A PCA of five standardized variables has eigenvalues \(1.40,1.20,0.90,0.80,0.70\). How many components are needed to retain at least 75%, and what proportion do they retain? (Answer: four components; \(86\%\).)
For the five eigenvalues in Question 4, a candidate reports that the first component explains \(1.40/1.20\) of the variance. Identify the error and give the correct proportion. (Answer: the denominator must be the total eigenvalue sum, 5; the correct proportion is \(1.40/5=28\%\).)

What You'll Learn