PCA Of Weakly Correlated Measurements

Why Weak Correlations Matter in PCA

Principal component analysis (PCA) can reduce many measurements to a smaller set of uncorrelated linear combinations. It is most effective for compression when a small number of components capture a large share of the total variance. When measurements are weakly correlated, however, there may be no direction that captures much more variation than the others. Variance can remain spread across several components, so retaining only one or two may discard substantial information.

The calculations below use standardized measurements and their correlation matrix. This is appropriate when the variables use different units or when the goal is to compare their variation on a common scale. Each standardized variable has variance 1. The eigenvalues of the correlation matrix give the variances of the principal-component scores, and their sum is the total variance.

Definition: For \(p\) standardized variables, let \(R\) be their \(p \times p\) correlation matrix. PCA finds orthogonal directions, represented by eigenvectors of \(R\). The corresponding eigenvalues, ordered as \(\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0\), are the variances of the component scores. The explained-variance ratio for component \(j\) is \(\lambda_j / \sum_{k=1}^{p}\lambda_k\); the cumulative ratio for the first \(q\) components is \(\sum_{j=1}^{q}\lambda_j / \sum_{k=1}^{p}\lambda_k\).

For a correlation matrix, every diagonal entry is 1, so its trace—the sum of its diagonal entries—is \(p\). The eigenvalues sum to the trace, making the total standardized variance \(p\). Thus, a component with eigenvalue 1 captures \(1/p\) of the total variance. An eigenvalue only slightly greater than 1 may represent a real direction of variation, but it is not necessarily enough to justify a strong reduction in dimension.

Worked Examples: Variance Spread Across Components

Worked Example: Two Weakly Correlated Measurements

A hypothetical risk team standardizes two measurements for each account: an annual change in reported claim count and an annual change in average claim cost. Their correlation is \(0.15\). Find the eigenvalues, the explained-variance ratios, and the proportion lost if only the first component is retained.

Method. Form the correlation matrix. For a two-variable correlation matrix with correlation \(r\), the directions proportional to \((1,1)\) and \((1,-1)\) have eigenvalues \(1+r\) and \(1-r\), respectively. Here:

R= \begin{pmatrix} 1 & 0.15\\ 0.15 & 1 \end{pmatrix}, \qquad \lambda_1=1+0.15=1.15, \qquad \lambda_2=1-0.15=0.85.

The total variance is \(1.15+0.85=2\), which also equals the sum of the two diagonal entries of \(R\). The explained-variance ratios are \(1.15/2=0.575\) and \(0.85/2=0.425\), or 57.5% and 42.5%.

Keeping only the first component therefore discards 42.5% of the total variance. The first component captures more than half, but not enough to summarize these two measurements with little loss. Check: the ratios add to 100%, and the eigenvalues add to the total variance of 2. The common mistake is to assume that two variables must compress well because their correlation is small; here the low correlation leaves nearly half the variance outside the first component.

Worked Example: Four Measurements With One Weak Association

Suppose four hypothetical standardized account indicators have correlation matrix

R= \begin{pmatrix} 1 & 0.20 & 0 & 0\\ 0.20 & 1 & 0 & 0\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1 \end{pmatrix}.

Find the ordered eigenvalues and decide how many components are needed to retain at least 75% of the variance.

The first two variables form a \(2 \times 2\) block with correlation \(0.20\). Its eigenvalues are \(1+0.20=1.20\) and \(1-0.20=0.80\). Each of the remaining variables is uncorrelated with the others, so each contributes an eigenvalue of 1. Ordered from largest to smallest, the eigenvalues are \(1.20, 1, 1, 0.80\). Their sum is 4, as required by the four diagonal entries.

The explained-variance ratios are \(1.20/4=30\%\), \(1/4=25\%\), \(1/4=25\%\), and \(0.80/4=20\%\). The cumulative proportions are 30%, 55%, 80%, and 100%. Two components fall short of 75%; three components reach 80%. Therefore, three components are required.

The two components associated with the eigenvalue 1 each capture 25%. Since those eigenvalues are tied, the particular orthogonal directions chosen within their two-dimensional subspace are not unique. This does not affect the cumulative variance. Check: the four ratios sum to 100%, and the first three eigenvalues sum to \(3.20\), which is 80% of the total 4. The trap is to count the weakly correlated pair as a single source of information and conclude that two components must suffice. The variance target, not the number of visible associations, determines the component count.

Worked Example: Testing a Retention Target for Three Measurements

Three hypothetical standardized measures of policy activity have the same pairwise correlation, \(0.10\). A report will retain components until at least 75% of the total variance is captured. Find the eigenvalues and the minimum number of components required.

The correlation matrix has 1 on its diagonal and \(0.10\) in every off-diagonal position. The direction \((1,1,1)\) has eigenvalue \(1+2(0.10)=1.20\). Any direction perpendicular to \((1,1,1)\) has eigenvalue \(1-0.10=0.90\); there are two independent such directions. The ordered eigenvalues are therefore \(1.20,0.90,0.90\).

Their sum is 3. The explained-variance ratios are \(1.20/3=40\%\), \(0.90/3=30\%\), and \(0.90/3=30\%\). The first component alone captures 40%; the first two capture \(40\%+30\%=70\%\). Because 70% is below the 75% target, two components do not meet the requirement. All three are needed to reach 100%.

This result is consistent with the weak pairwise correlations: the first component captures the shared direction, but that direction contains only 40% of the variance. The remaining 60% is distributed across two other directions. Check: the three ratios sum to 100%, and the first two eigenvalues sum to \(2.10\), which is 70% of 3. A common error is to round 70% up to a stated 75% threshold. A target is met only when the cumulative ratio is at least the target.

Why the Eigenvalues Measure Variance

The connection between eigenvalues and variance follows from how PCA rotates the standardized measurements. Let \(Z\) be a centered data vector with covariance matrix \(R\), and let \(a_j\) be a unit eigenvector of \(R\) with eigenvalue \(\lambda_j\). The score on that direction is \(a_j^\mathsf{T}Z\). Its variance is:

\operatorname{Var}(a_j^\mathsf{T}Z) = a_j^\mathsf{T}R a_j = a_j^\mathsf{T}(\lambda_j a_j) = \lambda_j a_j^\mathsf{T}a_j = \lambda_j.

The last equality uses \(a_j^\mathsf{T}a_j=1\). Since the component directions are orthonormal, changing from the original standardized coordinates to component coordinates is an orthogonal rotation. A rotation changes the directions in which variation is described, but not the total variation. Therefore, the component variances sum to the original total variance, \(\operatorname{tr}(R)=p\). This is why a cumulative explained-variance ratio measures the fraction of total variance retained.

This argument applies to the covariance matrix used for PCA. If PCA is performed on a covariance matrix of unstandardized variables, its eigenvalues still give component variances, but the total is the sum of the original variable variances rather than \(p\). The resulting proportions can be dominated by measurement scale. A correlation-matrix analysis instead gives each variable unit variance before the rotation.

Interpreting Weak Correlations Carefully

Weak pairwise correlations often indicate that no single linear combination captures a dominant share of the variation. They do not guarantee that every eigenvalue is exactly 1, or that the measurements are independent. Eigenvalues depend on the full joint correlation structure, including how correlations combine across variables. Small correlations can create a modestly larger first eigenvalue, as in the examples, while still leaving much variance in later components.

The practical question is not simply whether the first component has the largest eigenvalue. It is how much of the total variance the retained components capture, and whether that amount is acceptable for the intended use. A model that needs a compact representation may accept some loss; a risk analysis in which differences among measurements matter may require more components. The retention rule should be chosen for the purpose of the analysis, rather than inferred from the fact that the eigenvalues are ordered.

Observed pattern	What PCA indicates	Interpretive caution
All eigenvalues are close to 1 for standardized variables	Variance is spread fairly evenly across directions	A small number of components may preserve little of the total
The first eigenvalue is moderately above 1	The first direction captures more variance than an average standardized variable	Compare its ratio with the chosen retention target
Several early eigenvalues are similar	Several directions contribute meaningfully to total variance	Do not treat the first component as a complete summary

PCA is a linear variance-reduction method. A component with a low variance can still contain information relevant to a separate objective, such as prediction or interpretation; explained variance alone does not establish usefulness for that objective. Also, zero or weak correlation is not, by itself, evidence of independence. The component scores are uncorrelated, but uncorrelated scores need not be independent unless additional distributional conditions hold.

Key takeaway: With weakly correlated standardized measurements, variance may be spread across several principal components. Calculate the eigenvalues and cumulative explained-variance ratios, then apply the stated retention target. Do not assume that one or two components preserve most of the information merely because PCA orders them from largest to smallest.

Check Your Understanding

For each problem, calculate the relevant eigenvalues or cumulative explained-variance ratios before deciding how much variance is retained.

Two standardized hypothetical measurements have correlation \(-0.12\). Find both eigenvalues and the percentage of total variance captured by the first component. (Answer: eigenvalues \(1.12\) and \(0.88\); the first component captures 56.0000%.)
Three standardized hypothetical indicators have every pairwise correlation equal to \(0.05\). Find the eigenvalues and determine whether the first two components reach a 70% retention target. (Answer: eigenvalues \(1.10, 0.95, 0.95\); the first two capture \(2.05/3=68.3333\%\), so all three are needed for at least 70%.)
A four-variable correlation matrix has one pairwise correlation of \(0.10\), with all other off-diagonal entries zero. Find the ordered eigenvalues and the cumulative proportion for the first two components. (Answer: eigenvalues \(1.10, 1, 1, 0.90\); the first two capture \(2.10/4=52.5000\%\).)
A standardized three-variable analysis has eigenvalues \(1.16, 0.94, 0.90\). Does retaining the first two components meet a 70% target? State the proportion discarded if only those two are retained. (Answer: yes; the first two capture \(2.10/3=70.0000\%\), and the discarded proportion is 30.0000%.)
Three measurements are recorded in dollars, counts, and days, and their pairwise correlations are weak. The analysis aims to compare each measurement on a common scale. Which matrix should be used for PCA, and what is the key implication if the resulting eigenvalues are similar? (Answer: use the correlation matrix; similar eigenvalues indicate that variance is spread across components, so few components may not retain a large share.)

What You'll Learn