PCA When One Variable Dominates

Why One Variable Can Dominate PCA

Principal component analysis identifies directions of greatest variation. That makes the scale of the variables important: if one variable has much larger variance than the others, covariance-based PCA may devote its first component mainly to that variable. This can be appropriate when variation in the original units is the quantity of interest, but it can also reflect differences in measurement units rather than differences in analytical importance.

Standardizing each variable before PCA changes the question. Instead of comparing variation in the original units, the analysis compares variation relative to each variable’s own standard deviation. The resulting directions can be very different. The key is to choose the scale deliberately, then interpret the components and explained variance ratios on that scale.

Definition: Let \(x_{ij}\) be the value of variable \(j\) for observation \(i\), with \(n\) observations and \(p\) variables. Covariance-based PCA uses the sample covariance matrix \(S\) of the centered variables. Standardized PCA first subtracts each variable’s sample mean and divides by its sample standard deviation; it then uses the correlation matrix \(R\). If \(D\) is the diagonal matrix of sample standard deviations, then \(R=D^{-1}SD^{-1}\), provided each standard deviation is positive.

In covariance-based PCA, a principal component is a linear combination of centered variables. Its loading vector has length one, and the first component is chosen to have the greatest possible sample variance among such combinations. In standardized PCA, the same idea is applied to the standardized variables. Because standardization changes the variables’ scales, it also changes which directions have the greatest variance.

How Scale Changes the PCA Calculation

A covariance matrix records both the variance of each variable and the covariances between pairs of variables. Its diagonal entries are variances, so a very large diagonal entry can dominate the eigenvalue calculation. The eigenvector for the largest eigenvalue gives the first component’s loadings. The explained variance ratio for a component is its eigenvalue divided by the sum of all eigenvalues of the matrix being analyzed.

For correlation-based PCA, the standardized variables each have variance one, so the diagonal of \(R\) consists of ones. Off-diagonal entries are correlations, which are unchanged by measuring a variable in different units. Standardization therefore removes differences in marginal variance from the comparison, but it does not remove relationships among variables.

Key result: Covariance-based PCA answers which unit-length combination has the greatest variance in the original measurement units. PCA on standardized variables answers which unit-length combination has the greatest variance after each variable has been put on a standard-deviation scale. Neither scale is universally correct; the appropriate choice depends on the goal and the meaning of the measurements.

Worked Example: A High-Variance Variable Leads Covariance-Based PCA

A hypothetical risk study records two measurements for each policy segment: an exposure index and a service-delay measure. The sample covariance matrix, in the variables’ original units, is

S= \begin{pmatrix} 100 & 10\\ 10 & 4 \end{pmatrix}.

Find the first covariance-based component and its explained variance ratio. Then find the first component after standardization. The method is to compare the eigenvectors and eigenvalues of \(S\) and \(R\); the common mistake to avoid is assuming that standardization leaves the first component unchanged.

Covariance-based PCA. The eigenvalues solve \(\det(S-\lambda I)=0\). The trace is \(100+4=104\), and the determinant is \(100(4)-10(10)=300\). Thus

\lambda^2-104\lambda+300=0,\qquad \lambda_{1,2}=\frac{104\pm\sqrt{104^2-4(300)}}{2} =\frac{104\pm\sqrt{9616}}{2}.

Since \(\sqrt{9616}\approx98.0612\), the eigenvalues are \(\lambda_1\approx101.0306\) and \(\lambda_2\approx2.9694\). For the first eigenvector, the first row of \((S-\lambda_1 I)v=0\) gives \(-1.0306v_1+10v_2=0\), so \(v_2/v_1\approx0.1031\). Normalizing to length one gives approximately \(v_1=(0.9947,0.1025)^\mathsf{T}\). The first component is therefore mostly the exposure index. Its explained variance ratio is

\frac{101.0306}{101.0306+2.9694} =\frac{101.0306}{104} \approx0.9714.

After standardization. The standard deviations are \(10\) and \(2\), so the correlation is \(10/(10\cdot2)=0.5\). Therefore,

R= \begin{pmatrix} 1 & 0.5\\ 0.5 & 1 \end{pmatrix}.

The eigenvalues are \(1.5\) and \(0.5\); the first eigenvector is \((1,1)^\mathsf{T}/\sqrt{2}\). Its explained variance ratio is \(1.5/(1.5+0.5)=0.75\). Unlike the covariance-based component, this component weights the two standardized variables equally. The results check: the covariance eigenvalues sum to \(104\), the trace of \(S\), and the correlation eigenvalues sum to \(2\), the trace of \(R\).

The contrast arises because the first variable’s variance is \(25\) times the second variable’s variance. Covariance-based PCA reflects that difference; standardization does not. The correlation of \(0.5\) remains in the standardized analysis.

Why the Largest-Eigenvalue Direction Maximizes Variance

The connection between eigenvectors and component variance explains why a high-variance variable can control the first component. Let \(z\) be a centered observation vector with covariance matrix \(S\), and let \(a\) be a proposed loading vector with \(a^\mathsf{T}a=1\). The component score is \(a^\mathsf{T}z\). Its variance is

\operatorname{Var}(a^\mathsf{T}z) =E\left[(a^\mathsf{T}z)^2\right] =a^\mathsf{T}E[zz^\mathsf{T}]a =a^\mathsf{T}Sa.

The middle equality follows by expanding the scalar product and using the definition of the covariance matrix, \(S=E[zz^\mathsf{T}]\) for centered observations (or its sample counterpart). Since \(S\) is symmetric, write its eigenvalue decomposition as \(S=Q\Lambda Q^\mathsf{T}\), where the columns of \(Q\) are orthonormal eigenvectors and the diagonal entries of \(\Lambda\) are eigenvalues. Put \(b=Q^\mathsf{T}a\). Then \(b^\mathsf{T}b=1\), and

a^\mathsf{T}Sa=b^\mathsf{T}\Lambda b =\sum_{j=1}^p\lambda_j b_j^2 \leq \lambda_1\sum_{j=1}^p b_j^2 =\lambda_1,

where \(\lambda_1\) is the largest eigenvalue. Equality holds when \(b\) has all its weight in an eigendirection for \(\lambda_1\), so the maximizing loading vector is an eigenvector associated with \(\lambda_1\). If that eigenvalue is repeated, more than one maximizing direction may exist. This result applies to the covariance matrix used: replacing \(S\) with \(R\) changes the matrix and therefore may change the maximizing direction.

Worked Example: A Three-Variable Scale Change

A hypothetical portfolio review measures a large-scale balance indicator \(X_1\) and two operational indicators \(X_2\) and \(X_3\). Suppose the covariance matrix is

S= \begin{pmatrix} 100&0&0\\ 0&4&2\\ 0&2&4 \end{pmatrix}.

Find the leading component and explained variance ratio for covariance-based PCA, then repeat the comparison after standardization. The common mistake is to assume that the leading component must continue to involve \(X_1\) simply because its variance is largest in the original units.

Covariance-based result. The first coordinate is independent of the lower \(2\) by \(2\) block for the eigenvalue calculation, so one eigenvalue is \(100\). The lower block has characteristic equation

\det\begin{pmatrix}4-\lambda&2\\2&4-\lambda\end{pmatrix} =(4-\lambda)^2-4=0.

Thus its eigenvalues are \(6\) and \(2\), with corresponding directions \((1,1)^\mathsf{T}/\sqrt{2}\) and \((1,-1)^\mathsf{T}/\sqrt{2}\) in the \(X_2,X_3\) coordinates. The full eigenvalues are \(100,6,2\), so the first covariance-based component is \(X_1\) alone. The total variance is \(100+6+2=108\), giving first-component explained variance ratio \(100/108\approx0.9259\).

Standardized result. Each variable’s standard deviation is respectively \(10,2,2\). Dividing each covariance by the product of its standard deviations gives

R= \begin{pmatrix} 1&0&0\\ 0&1&0.5\\ 0&0.5&1 \end{pmatrix}.

The lower block now has eigenvalues \(1.5\) and \(0.5\); the first variable contributes the separate eigenvalue \(1\). The largest eigenvalue is \(1.5\), with loading vector \((0,1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\). The total standardized variance is \(3\), so its explained variance ratio is \(1.5/3=0.5\). The check is that the eigenvalues sum to \(3\), the trace of the correlation matrix. In this analysis the leading direction combines the two operational indicators, while the high-variance balance indicator is not part of that direction.

Choosing the Scale for the Question

The difference between these analyses is not a defect in PCA. They optimize variance on different scales. Covariance-based PCA can be suitable when the original units are comparable and their variance differences are meaningful for the objective. For example, an analysis may intentionally prioritize variation in a monetary measurement over variation in a small-range operational score. In that case, the large variance is part of the intended weighting.

Standardized PCA is often more suitable when variables use different units or when no variable should receive greater influence solely because its numerical scale is larger. It gives each variable variance one before accounting for its correlations with the others. That does not make all variables equally important in every practical sense: it makes their initial marginal scales equal for the PCA calculation.

Consider a hypothetical claims dataset containing annual claim count and total claim amount measured in thousands of dollars. The variables have different units and may have very different variances. Covariance-based PCA can be driven largely by total claim amount, while standardized PCA compares movement in each variable relative to its own standard deviation. The choice depends on whether raw-unit variation or relative variation is the intended basis for summarizing the data.

Worked Example: Interpreting a Strong Correlation at Different Scales

For a hypothetical two-variable analysis, let \(X_1\) be an annual severity measure in thousands of dollars and \(X_2\) a count measure. Suppose their variances are \(100\) and \(1\), and their correlation is \(0.8\). Compare the first component’s direction and explained variance ratio under covariance-based and standardized PCA. The method is to recover the covariance from the correlation, then analyze both matrices; the common mistake is to infer equal influence from a high correlation alone.

The standard deviations are \(10\) and \(1\). Since the covariance is correlation times the product of standard deviations, it is \(0.8(10)(1)=8\). Thus

S= \begin{pmatrix} 100&8\\ 8&1 \end{pmatrix}.

Its trace is \(101\) and determinant is \(100(1)-8^2=36\). The eigenvalues are

\lambda_{1,2} =\frac{101\pm\sqrt{101^2-4(36)}}{2} =\frac{101\pm\sqrt{10057}}{2} \approx100.6423,\;0.3577.

For the leading eigenvector, the first row gives \((100.6423-100)v_1=8v_2\), so \(v_2/v_1\approx0.0803\). After normalizing, the loading vector is approximately \((0.9968,0.0800)^\mathsf{T}\). The covariance-based component is almost entirely the severity measure, and its explained variance ratio is \(100.6423/101\approx0.9965\).

After standardization, the matrix is \(R=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\). Its eigenvalues are \(1.8\) and \(0.2\), with leading loading vector \((1,1)^\mathsf{T}/\sqrt{2}\). The explained variance ratio is \(1.8/2=0.9\). Both analyses reflect the positive relationship, but the covariance-based analysis is dominated by the larger variance of \(X_1\). The eigenvalues also check: they sum to \(101\) for \(S\) and to \(2\) for \(R\).

Common Mistakes and Practical Checks

Choosing a scale without stating the reason. Before computing PCA, identify whether the analysis should preserve original-unit variation or compare variables relative to their own scales.
Assuming standardization merely changes the eigenvalues. In general, \(R=D^{-1}SD^{-1}\), which can change both eigenvalues and eigenvectors.
Interpreting a large covariance-based loading as proof of greater importance. A large variance may arise because a variable is measured in large numerical units. Loadings describe the chosen PCA scale, not importance in every substantive sense.
Comparing explained variance ratios across scales as if they were the same quantity. Each ratio is relative to the total variance in its own matrix. A covariance-based ratio and a standardized ratio answer different questions.
Standardizing a constant variable. If a variable has zero standard deviation, division by that standard deviation is undefined. It must be addressed before correlation-based PCA.

As a final check, eigenvalues should be nonnegative for a valid covariance or correlation matrix, and their sum should equal the matrix trace. The first component’s eigenvalue should be at least as large as every other eigenvalue. These checks can reveal arithmetic or interpretation errors, but they do not determine which scale is appropriate.

Key takeaway: When one variable dominates covariance-based PCA, inspect its variance, units, and purpose in the analysis. Compare with standardized PCA when variables are on different scales, and explain which variation the chosen components summarize. This scale-aware interpretation is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each problem, state which scale is being used and check that the eigenvalues add to the matrix trace when a matrix is supplied.

A covariance matrix is \(\begin{pmatrix}49&0\\0&1\end{pmatrix}\). Identify the covariance-based leading direction and explained variance ratio. If both variables are standardized, what is the correlation matrix, and is its leading direction unique? (Answer: Covariance PCA has direction \((1,0)^\mathsf{T}\) and ratio \(49/50=0.9800\); the standardized matrix is \(I\), whose tied eigenvalues make the leading direction non-unique.)
Two variables have standard deviations \(5\) and \(2\), with correlation \(0.5\). Write their correlation matrix, find the leading standardized loading vector, and calculate its explained variance ratio. (Answer: \(R=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\); leading vector \((1,1)^\mathsf{T}/\sqrt{2}\), eigenvalue \(1.5\), ratio \(0.7500\).)
Suppose a hypothetical dataset has two variables with variances \(25\) and \(4\), and covariance \(5\). Find their correlation and the leading standardized component’s explained variance ratio. (Answer: Correlation \(5/(5\cdot2)=0.5\); the standardized leading component has ratio \(0.7500\).)
A hypothetical study measures annual severity with standard deviation \(12\) and event count with standard deviation \(2\); their correlation is \(-0.25\). Give the leading standardized loading vector and explained variance ratio. (Answer: \(R=\begin{pmatrix}1&-0.25\\-0.25&1\end{pmatrix}\); leading vector \((1,-1)^\mathsf{T}/\sqrt{2}\), eigenvalue \(1.25\), ratio \(1.25/2=0.6250\).)
A covariance matrix is \(\begin{pmatrix}64&0&0\\0&9&3\\0&3&9\end{pmatrix}\). Find the leading covariance-based component and its explained variance ratio. Then standardize and identify the leading standardized direction and ratio. (Answer: Covariance eigenvalues are \(64,12,6\), so the leading direction is \((1,0,0)^\mathsf{T}\) with ratio \(64/82\approx0.7805\). The standardized matrix has lower-block correlation \(3/9=1/3\), eigenvalues \(4/3,1,2/3\); its leading direction is \((0,1,1)^\mathsf{T}/\sqrt{2}\) with ratio \((4/3)/3=0.4444\).)

What You'll Learn