Why an Eigenvalue Tie Changes the PCA Answer
PCA is often presented as a ranked list of directions: the first principal component captures the most variance, the second captures the most remaining variance, and so on. When two or more eigenvalues are equal, however, that list does not determine a unique direction for each tied component. It determines a subspace, within which many equally valid orthonormal bases can be chosen.
This distinction matters when interpreting loadings, comparing PCA results from different software runs, or reducing the number of variables. The directions reported for a tied group can differ even though the underlying variance structure is the same. If a retention rule keeps only part of a tied group, even the retained subspace—and therefore individual observations’ reduced representations—can depend on the choice of basis.
The covariance matrix used in ordinary PCA is symmetric and positive semidefinite. Its eigenvalues are therefore real and nonnegative, and eigenvectors for distinct eigenvalues can be chosen orthogonal. For a repeated eigenvalue, the individual orthogonal eigenvectors are not uniquely determined: rotating them within their shared eigenspace produces another valid set of directions.
A Rotation Within a Tied Eigenspace
Suppose the covariance matrix has eigenvalue \(\lambda\) repeated \(r\) times, and let the columns of \(U\) be an orthonormal basis for that eigenspace. For any \(r\)-by-\(r\) orthogonal matrix \(Q\), the columns of \(UQ\) are another orthonormal basis for the same space. An orthogonal matrix satisfies \(Q^TQ=I\).
Thus the rotated columns remain eigenvectors with the same eigenvalue, and remain orthonormal. The choice of \(Q\) can be any rotation or reflection within the tied coordinates; it does not change the space spanned by the columns.
Worked Example: Rotating Two Directions With Equal Variance
A hypothetical centered portfolio dataset has covariance matrix \[ \Sigma= \begin{pmatrix} 5&0&0\\ 0&2&0\\ 0&0&2 \end{pmatrix}. \] Find two valid bases for the tied eigenspace and verify their variances and covariance.
Method. The eigenvectors \(e_2=(0,1,0)^T\) and \(e_3=(0,0,1)^T\) have eigenvalue \(2\). Rotate those vectors by 30 degrees, using \(\cos 30^\circ=\sqrt{3}/2\) and \(\sin 30^\circ=1/2\): \[ q_1=(0,\sqrt{3}/2,1/2)^T,\qquad q_2=(0,-1/2,\sqrt{3}/2)^T. \]
Check orthonormality. Each vector has squared length \(3/4+1/4=1\). Their inner product is \((\sqrt{3}/2)(-1/2)+(1/2)(\sqrt{3}/2)=0\). They form an orthonormal pair.
Check the eigenvalue. Multiplying either vector by \(\Sigma\) multiplies its second and third coordinates by 2 and leaves its first coordinate zero. Therefore \(\Sigma q_1=2q_1\) and \(\Sigma q_2=2q_2\). Each direction has variance \(2\), and their covariance is \(q_1^T\Sigma q_2=2q_1^Tq_2=0\).
The original basis \(e_2,e_3\) and the rotated basis \(q_1,q_2\) are equally valid. The common mistake is to treat the software’s first reported vector in this tied pair as uniquely determined. The correction is to report that the two-dimensional eigenspace is identified, while its particular basis is not.
What Remains Invariant
An orthonormal basis change alters the coordinates used to describe a subspace, but not the subspace itself. If \(U\) contains an orthonormal basis for a tied eigenspace, its projection matrix is \(UU^T\). For a rotated basis \(UQ\), the projection matrix is unchanged:
Consequently, projecting a centered observation onto the entire tied eigenspace gives the same vector whichever orthonormal basis is used. Its coordinates within that space can change, but the projection itself does not. The variance associated with the entire tied space is also invariant: if its dimension is \(r\), it is \(r\lambda\). The eigenvalue list, total variance, and proportion of total variance explained by the full tied group are unchanged.
The same reasoning clarifies the status of component scores. If a centered observation is \(x\), its coordinates in the basis \(U\) are \(U^Tx\). In the rotated basis they are \(Q^TU^Tx\). These coordinates can differ, while their squared length is preserved because \(Q\) is orthogonal. Therefore, do not interpret a change in individual tied-component scores as a change in the observation’s projection onto the full tied subspace.
Worked Example: Same Projection, Different Scores
A hypothetical two-variable dataset has covariance matrix \(\Sigma=4I\), where \(I\) is the two-by-two identity matrix. Compare the PCA scores and reconstructions for the centered observation \(x=(2,1)^T\), first using the coordinate basis and then a 45-degree rotated basis.
Method. Both eigenvalues are 4, so the entire two-dimensional space is tied. In the coordinate basis \(e_1=(1,0)^T,e_2=(0,1)^T\), the scores are \(e_1^Tx=2\) and \(e_2^Tx=1\). Keeping both components reconstructs \(2e_1+1e_2=(2,1)^T\).
For the rotated basis, take \(q_1=(1/\sqrt{2},1/\sqrt{2})^T\) and \(q_2=(-1/\sqrt{2},1/\sqrt{2})^T\). The scores are \[ q_1^Tx=(2+1)/\sqrt{2}=3/\sqrt{2},\qquad q_2^Tx=(-2+1)/\sqrt{2}=-1/\sqrt{2}. \] Reconstructing from both rotated scores gives \[ (3/\sqrt{2})q_1+(-1/\sqrt{2})q_2 =(3/2,3/2)^T+(1/2,-1/2)^T =(2,1)^T. \]
The two score pairs, \((2,1)\) and \((3/\sqrt{2},-1/\sqrt{2})\), differ, but both have squared length \(5\), and both reconstruct the same observation when the full tied space is retained. The trap is to conclude from different score coordinates that the PCA subspace or reconstruction must have changed.
A Derivation: Why Every Direction in the Tie Has the Same Variance
The variance of a centered observation projected onto a unit direction \(v\) is \(v^T\Sigma v\). This follows from the covariance definition: the scalar projection is \(v^Tx\), whose variance is \(v^T\operatorname{Cov}(x)v=v^T\Sigma v\). Now suppose \(v\) lies in the eigenspace for \(\lambda\), so \(\Sigma v=\lambda v\), and \(v^Tv=1\). Then
The same calculation shows why distinct orthogonal directions within the tie have zero covariance. If \(u\) and \(v\) are orthogonal eigenvectors with the same eigenvalue \(\lambda\), then \[ \operatorname{Cov}(u^Tx,v^Tx)=u^T\Sigma v =u^T(\lambda v)=\lambda u^Tv=0. \] Every orthonormal direction in the tied eigenspace therefore has variance \(\lambda\), and any orthonormal basis of it gives uncorrelated component scores with that variance.
When a Retention Cutoff Splits a Tie
A crucial qualification arises when PCA retains fewer directions than the dimension of a tied eigenspace. If the cutoff keeps the entire tied group, the retained subspace is well-defined. If it keeps only some of the tied directions, the selected subspace is not unique. Total retained variance can still be the same, while the reconstruction of a particular observation changes.
Worked Example: Retaining Only Part of a Tied Group
A hypothetical centered four-feature dataset has covariance matrix \(\Sigma=\operatorname{diag}(7,3,3,1)\). A two-component representation is requested. Compare two valid choices of the second direction and their reconstructions of \(x=(1,2,1,3)^T\).
Method. The first direction is \(e_1\), with eigenvalue 7. The next two eigenvalues are tied at 3, but only one direction from that tied pair can be retained. One choice is \(e_2\); another is \(q=(e_2+e_3)/\sqrt{2}\). Both are unit eigenvectors with eigenvalue 3 and orthogonal to \(e_1\).
With \(e_1,e_2\), the scores are \(e_1^Tx=1\) and \(e_2^Tx=2\), giving reconstruction \(1e_1+2e_2=(1,2,0,0)^T\). The squared reconstruction error is \[ \|(1,2,1,3)-(1,2,0,0)\|^2=1^2+3^2=10. \]
With \(e_1,q\), the second score is \(q^Tx=(2+1)/\sqrt{2}=3/\sqrt{2}\). Its contribution is \((3/\sqrt{2})q=(0,3/2,3/2,0)^T\). The reconstruction is therefore \((1,3/2,3/2,0)^T\), and the squared error is \[ (2-3/2)^2+(1-3/2)^2+3^2 =1/4+1/4+9=9.5. \]
Both choices retain variance \(7+3=10\) out of total variance \(7+3+3+1=14\), a proportion \(10/14=5/7\). Yet their reconstructions differ, as do their errors for this particular observation. The common mistake is to infer that equal retained variance guarantees identical observation-level reconstructions. It guarantees equal variance captured across the dataset, not an identical projection for every observation when the cutoff divides a tie.
Interpreting and Reporting Tied PCA Results
In practical work, exact ties may occur because of the structure of the data; near-ties may occur because two estimated eigenvalues are very close. An exact tie has the mathematical nonuniqueness described above. A near-tie is not an exact tie, but the corresponding directions can be sensitive to small changes in the data, so avoid treating a small numerical difference as strong evidence that one direction has a stable, distinct interpretation.
A PCA report should distinguish the eigenspace-level conclusion from the basis-level output. For an exact tie, report the tied eigenvalue and its multiplicity, and describe the span of the tied directions or the variables defining that span. If all tied directions are retained, discuss their combined variance and projection. Avoid presenting one direction’s loading pattern as uniquely meaningful merely because a particular calculation returned it first.
If a retention rule splits a tied group, state that the chosen lower-dimensional subspace is one of several equally ranked choices. Where the application requires a reproducible representation, use a clearly documented convention for selecting a basis and recognize that the convention does not make the selected basis statistically unique. If the scientific or business interpretation depends on a particular direction, investigate whether that direction is supported by information beyond the PCA variance ranking.
Inspect the eigenvalues and identify repeated or practically indistinguishable values.
Determine whether the selected components include all, none, or only part of the tied group.
For a fully included group, emphasize its eigenspace, projection, and combined variance; if the cutoff splits the group, disclose the resulting nonuniqueness.
The central exam distinction is between a direction and the space spanned by directions. A repeated eigenvalue makes the individual directions within its eigenspace nonunique. It does not make the eigenvalue, the full eigenspace, or the variance captured by that full eigenspace nonunique. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, identify the basis-dependent and invariant quantities before giving your answer.
- A covariance matrix has eigenvalues \(6,2,2\). What is the variance along any unit direction in the two-dimensional eigenspace for eigenvalue 2? (Answer: 2.)
- A tied eigenspace has an orthonormal basis matrix \(U\), and \(Q\) is orthogonal. State the projection matrix using the rotated basis \(UQ\), and compare it with the original. (Answer: \((UQ)(UQ)^T=UU^T\); the projection matrix is unchanged.)
- A dataset has eigenvalues \(8,4,4,1\), and both tied directions are retained along with the first component. How much variance is retained, and what is the total variance? (Answer: retained variance \(=8+4+4=16\); total variance \(=17\).)
- For eigenvalues \(9,3,3,1\), a two-component rule retains the first component and one direction from the tied pair. Are the retained variance and the retained subspace both unique? (Answer: retained variance is \(12\), but the retained subspace is not unique.)
- A centered observation has score coordinates \((a,b)\) in one orthonormal basis of a fully retained two-dimensional tied eigenspace. Its coordinates in another basis are \(Q^T(a,b)^T\), with \(Q\) orthogonal. What happens to the squared score length and the projection onto the full space? (Answer: the squared score length \(a^2+b^2\) and the full-space projection are unchanged; the individual coordinates may change.)