Why Nearly Tied Eigenvalues Need Careful Interpretation
Principal components are directions of greatest variance, ordered by their eigenvalues. When the leading eigenvalue is clearly larger than the next one, the leading direction is usually identifiable: a small change in the covariance matrix tends to make only a small change in that direction. When two eigenvalues are close, the same-sized change can rotate their individual directions substantially. This matters when interpreting loadings or component scores: a reported direction may depend sensitively on the particular observations used to calculate the matrix.
The sensitivity of individual directions does not mean that every PCA conclusion is unstable. If a group of nearly tied components is separated from the remaining components, the subspace spanned by that group can remain stable even while the axes chosen within it rotate. The group’s total captured variance can also remain nearly unchanged. The important distinction is between the orientation of an individual axis and the orientation and variance of a whole component subspace.
Here, a perturbation means a change \(\Delta\) in the matrix, giving a new matrix \(S+\Delta\). It can arise from sampling different observations, a small measurement change, or a revised data set. The comparison assumes the same variables and the same covariance-or-correlation choice in both analyses. A change of measurement scale or a switch between covariance and correlation PCA is a different change in the analysis, not merely a small matrix perturbation.
A Two-Variable Derivation of Direction Sensitivity
The mechanism is especially clear for a symmetric two-variable matrix. Write it as \(S=\begin{pmatrix}a&b\\b&d\end{pmatrix}\). A unit direction making angle \(\theta\) with the first coordinate axis is \(u=(\cos\theta,\sin\theta)\). Its variance is the Rayleigh quotient \(u^\mathsf{T}Su\). The leading principal direction maximizes this variance.
Expanding that variance gives \(Q(\theta)=a\cos^2\theta+2b\sin\theta\cos\theta+d\sin^2\theta\). At an eigenvector direction its derivative is zero. Differentiating and using the double-angle identities gives the condition below.
This formula shows why the gap matters. When \(a-d\) is small, a modest change to the off-diagonal entry \(b\) can make the ratio large and the angle change substantially. The formula determines an eigenvector orientation modulo the usual axis ambiguity; choosing the leading rather than the trailing direction is done by checking which direction has the larger eigenvalue. If the matrix is exactly a multiple of the identity, every direction has the same variance, so no unique principal direction exists.
Worked Example: A Small Off-Diagonal Change Rotates the Leading Direction
A two-variable PCA initially has covariance matrix \(S_0=\begin{pmatrix}4.02&0\\0&3.98\end{pmatrix}\). A revised data set gives \(S_1=\begin{pmatrix}4.02&0.03\\0.03&3.98\end{pmatrix}\). Find the leading direction for each matrix, the new eigenvalues, and explain the effect of the perturbation.
Method. Use the diagonal entries to identify the original directions, then apply the two-variable angle formula and eigenvalue formula to the revised matrix. The perturbation changes only the off-diagonal entries, by 0.03 each.
For \(S_0\), the eigenvalues are 4.02 and 3.98, so the leading direction is \(u_1=(1,0)\). The gap is \(4.02-3.98=0.04\).
For \(S_1\), \(a-d=4.02-3.98=0.04\) and \(2b=0.06\). Therefore,
The revised eigenvalues are the mean of the diagonal entries plus or minus the square root of the squared half-gap and squared off-diagonal entry:
Thus the eigenvalues are approximately \(4.03606\) and \(3.96394\). The leading direction has rotated about \(28.15^\circ\), even though the matrix entries changed by only 0.03. The calculation is checked by the eigenvalue sum: \(4.03606+3.96394=8.00000\), equal to the trace \(4.02+3.98=8.00\). The direction changed substantially because the original gap, 0.04, was small relative to the perturbation. A common mistake is to call the new direction “almost unchanged” just because the matrix entries changed by small absolute amounts; the gap must also be considered.
When the Directions Move but the Subspace Holds
Suppose two components are retained together. If their eigenvalues are close to one another but both are well separated from the next eigenvalue, their individual axes can rotate within their shared plane without much change to that plane. In that case the two-dimensional subspace is a more stable object to interpret than either component direction separately. A vector’s projection onto the plane, and the total variance represented by the plane, can remain stable even when the coordinates along the individual axes change.
For a fixed matrix, the sum of the eigenvalues for a retained set of components is the total variance represented by those components. If the retained set is exactly a two-dimensional invariant subspace, rotating the orthonormal axes within that plane does not change the plane or the sum of its two eigenvalues. With a perturbed matrix, the sum need not be exactly the same, but it can remain close when the perturbation is small and the subspace is well separated from the rest of the spectrum.
Worked Example: A Nearly Tied Pair With a Stable Two-Dimensional Subspace
Consider \(S=\begin{pmatrix}5.01&0.02&0\\0.02&4.99&0\\0&0&1.00\end{pmatrix}\). Find the angle of the leading direction in the first two coordinates, the two leading eigenvalues, and the variance and subspace represented by the first two components.
Method. Apply the two-variable formulas to the upper-left block. Because the third coordinate has no covariance with the first two, it is a separate eigenvector direction, \((0,0,1)\), with eigenvalue 1.00.
For the upper-left block, \(a-d=5.01-4.99=0.02\) and \(2b=0.04\). Hence \(\tan(2\theta)=0.04/0.02=2\), so \(\theta=\frac{1}{2}\arctan(2)\approx31.72^\circ\).
The mean of the two diagonal entries is \((5.01+4.99)/2=5.00\), and the square-root term is \(\sqrt{0.01^2+0.02^2}=\sqrt{0.0005}\approx0.02236\). The leading eigenvalues are therefore approximately 5.02236 and 4.97764. The first direction has rotated substantially from the original first coordinate axis, but both leading directions remain in the first-two-coordinate plane.
The sum of the leading eigenvalues is \(5.02236+4.97764=10.00000\). The full trace is \(5.01+4.99+1.00=11.00\), so the first two components represent \(10/11\approx0.9091\), or 90.91%, of the total variance. As a check, the eigenvalue sum is also the trace of the upper-left block, \(5.01+4.99=10.00\). Here the pair’s shared subspace and combined variance are clear, while the exact orientation of its first axis is sensitive to the small gap. The common mistake is to interpret a changed first component as proof that the whole two-component representation has changed equally; the plane itself has not changed in this example.
What Changes in Component Scores?
A score is the projection of a centered observation onto a component direction. If the directions rotate within a retained subspace, the individual scores generally change because each score uses a different axis. However, the set of scores across the retained axes still describes the same projection when the subspace itself is unchanged. In particular, the sum of squared scores for an observation equals the squared length of its projection onto that subspace, regardless of which orthonormal axes are used within it.
Thus, when two eigenvalues are nearly tied, interpretation should not overstate the importance of which of the two axes is called first. A loading pattern or score attributed to one direction may be sensitive to small data changes; the combined pattern across the two directions may be more informative. This does not mean that individual scores are meaningless in every application. It means their stability should be assessed relative to the eigenvalue gap and the purpose of the interpretation.
Worked Example: Rotated Scores Preserve the Projection Onto the Pair
A covariance matrix is \(S=\begin{pmatrix}8.01&0.01&0\\0.01&7.99&0\\0&0&2.00\end{pmatrix}\), and a centered observation is \(x=(2,1,3)\). Find the leading direction’s angle, the first two component scores, and check the squared-score total against the projection onto the retained subspace.
Method. The first two coordinates form an invariant plane, so first find the rotated orthonormal axes in that plane. Then project \(x\) onto each axis and compare the sum of squared scores with the squared length of the two-coordinate projection.
For the upper-left block, \(a-d=8.01-7.99=0.02\) and \(2b=0.02\). Therefore, \(\tan(2\theta)=1\) and \(\theta=22.5^\circ\). The first two unit directions are \(u_1=(\cos22.5^\circ,\sin22.5^\circ,0)\) and \(u_2=(-\sin22.5^\circ,\cos22.5^\circ,0)\).
Using \(\cos22.5^\circ\approx0.9239\) and \(\sin22.5^\circ\approx0.3827\), the scores are
Their squared-score total is approximately \(2.2304^2+0.1585^2\approx4.9747+0.0251=4.9998\), which rounds to 5.0000. The projection onto the first-two-coordinate plane is \((2,1,0)\), with squared length \(2^2+1^2=5\). The tiny difference before rounding comes from rounding the trigonometric values. The third coordinate, 3, is excluded from this retained subspace. The check confirms that the rotated score pair preserves the projected length, even though the individual scores differ from the unrotated coordinates 2 and 1. A common mistake is to compare a score on a rotated axis directly with an old coordinate and expect them to agree.
Reading Stability Without Overclaiming
The gap is a sensitivity warning, not a complete measure of every practical uncertainty. For a single leading direction, compare its eigenvalue with the next one. For a group of retained directions, compare the group with the eigenvalues just outside it: a clear boundary gap supports a stable subspace, even if there are small gaps within the group. In a larger PCA, the same principle applies to a cluster of several nearly tied eigenvalues.
A useful examination sequence is to identify the relevant gap, determine whether the perturbation is being compared with that gap, and then state which object is stable: an individual direction, a group subspace, or the variance captured by that group. In actual data analysis, a small gap alone does not tell you the size of plausible data perturbations; that depends on the data and how the analysis is conducted. Avoid claiming a direction must be unstable without considering the perturbation, or that every component result is unstable merely because two eigenvalues are close.
Check Your Understanding
For each problem, distinguish sensitivity of individual directions from stability of a retained subspace where relevant.
- A two-variable matrix has \(a=3.02\), \(d=2.98\), and \(b=0.01\). Find the leading direction’s angle from the first coordinate axis. (Answer: approximately \(13.28^\circ\).)
- A two-variable matrix has \(a=7.01\), \(d=6.99\), and \(b=0.03\). Find its two eigenvalues and the leading direction angle. (Answer: eigenvalues approximately 7.0316 and 6.9684; angle approximately \(35.78^\circ\).)
- A three-variable matrix has upper-left block \(\begin{pmatrix}4.005&0.005\\0.005&3.995\end{pmatrix}\), zero covariances with the third variable, and third diagonal entry 1. What is the first-two-component subspace, and what is the sum of its eigenvalues? (Answer: the first-two-coordinate plane; eigenvalue sum 8.000.)
- A two-component subspace is unchanged by a rotation of its orthonormal axes. An observation has projection \((3,4)\) onto that subspace. What is the sum of its two squared component scores before and after the rotation? (Answer: 25 in both cases.)
- Two leading eigenvalues are nearly equal, but both are much larger than the third eigenvalue. Which is generally more defensible to interpret as stable: the first eigenvector alone or the two-dimensional subspace? (Answer: the two-dimensional subspace, provided the perturbation is small relative to its separation from the third eigenvalue.)