Why Equal Eigenvalues Change the Meaning of an Axis
A principal component is often described as a direction that captures a particular amount of variance. When two or more covariance eigenvalues are equal, however, the variance does not distinguish directions within their shared eigenspace. The individual axes reported by a PCA calculation can then depend on the particular orthonormal basis returned by the calculation, even though the subspace they span is determined.
This is more than the sign ambiguity of a single eigenvector. If an eigenvalue is repeated, axes can be rotated into one another while remaining valid principal directions. It is essential to distinguish three things: the eigenspace, a chosen set of axes that spans it, and the reconstruction obtained by projecting onto the whole space.
Write \(V\) for a matrix whose columns are orthonormal eigenvectors of \(\Sigma\). For a centered observation \(x\), the coordinates, or scores, in these axes are \(V^\top x\), and the projection reconstruction is \(VV^\top x\). If the axes span only a selected collection of principal directions, \(V\) contains just those columns. The reconstruction uses the subspace they span, not the names or order assigned to its basis vectors.
Rotating a Complete Repeated-Eigenvalue Block
Suppose \(V\) has \(m\) orthonormal columns spanning \(E_\lambda\). Then \(\Sigma V=\lambda V\). Choose any \(m\times m\) orthogonal matrix \(Q\), meaning \(Q^\top Q=I_m\), and set \(W=VQ\). The columns of \(W\) are another orthonormal basis of the same eigenspace.
Proof. Since \(Q\) is orthogonal, \(W^\top W=Q^\top V^\top VQ=Q^\top Q=I_m\), so the columns of \(W\) are orthonormal. Also,
Thus each column remains an eigenvector with eigenvalue \(\lambda\). Finally,
which proves equality of the projection matrices. \(\square\)
The same invariance holds for the covariance contribution reconstructed from the full block. Within this block the eigenvalue matrix is \(\lambda I_m\), so
The scores themselves need not be equal: \(W^\top x=Q^\top V^\top x\). They are rotated coordinates for the same projection. A report that gives individual scores or axis loadings can therefore differ between valid PCA outputs, even when the full tied block reconstructs every centered observation identically.
Worked Example: Rotating a Two-Dimensional Tied Block
Let \(\Sigma=\operatorname{diag}(9,4,4,1)\). The second and third coordinate axes span the eigenspace for the repeated eigenvalue \(4\). Choose the original basis \(V=(e_2,e_3)\), and rotate it by 45 degrees:
Both columns of \(W\) are orthonormal. Because \(\Sigma e_2=4e_2\) and \(\Sigma e_3=4e_3\), linearity gives \(\Sigma w=4w\) for each rotated column \(w\). The two bases are visibly different, but their projection matrix is
For any centered observation \(x=(x_1,x_2,x_3,x_4)^\top\), either basis reconstructs the tied-block part as \((0,x_2,x_3,0)^\top\). In the original basis the scores are \((x_2,x_3)^\top\); in the rotated basis they are \(((x_2+x_3)/\sqrt{2},(-x_2+x_3)/\sqrt{2})^\top\). The scores change, but the reconstructed vector does not. The covariance contribution from this block is \(4\operatorname{diag}(0,1,1,0)\) in either basis.
What If a Rank-\(K\) Truncation Cuts Through a Tie?
The invariance just proved applies when the entire repeated-eigenvalue block is included. A different issue arises if a rank-\(K\) approximation includes only part of a tied block. The selected axes still have the same eigenvalue, but different choices of axes can span different subspaces and produce different projections of particular observations.
To see why each choice is equally good for the PCA variance criterion, suppose all eigenspaces with eigenvalues strictly greater than \(\lambda\) are included, and \(r\) additional directions must be chosen from \(E_\lambda\). For any orthonormal \(r\)-column matrix \(U\) in \(E_\lambda\), \(\Sigma U=\lambda U\). The variance captured by these \(r\) directions is
It is the same for every such \(U\). In the rank-\(K\) PCA criterion, the higher-eigenvalue directions contribute the same amount whichever tied directions are chosen; hence every such choice has the same total captured variance and the same optimal reconstruction error. This does not imply that all choices reconstruct each observation in the same way: their projection matrices can differ.
Worked Example: Two Equally Good Rank-One Choices
Take \(\Sigma=5I_2\). Every unit vector is an eigenvector with eigenvalue \(5\), so either \(u_1=(1,0)^\top\) or \(u_2=(0,1)^\top\) is a valid first principal axis. For \(x=(2,1)^\top\), the rank-one projection reconstructions are
These are different reconstructions of this particular observation. Yet each direction captures variance \(u_j^\top\Sigma u_j=5\), and each leaves expected residual variance \(\operatorname{tr}(\Sigma)-5=10-5=5\). Neither axis is preferred by the covariance matrix. If both axes are included instead, the projection is \(I_2\), regardless of the orthonormal basis chosen, and the reconstruction of \(x\) is exactly \(x\).
Worked Example: Same Full-Block Reconstruction in New Coordinates
Consider \(\Sigma=\operatorname{diag}(6,2,2)\), and focus on the tied eigenspace spanned by \(e_2,e_3\). Let \(x=(1,3,-1)^\top\). In the standard basis for this block, its scores are \((3,-1)^\top\), and the block reconstruction is
Use instead the rotated basis \(w_1=(e_2+e_3)/\sqrt{2}\), \(w_2=(-e_2+e_3)/\sqrt{2}\). The new scores are \(w_1^\top x=(3-1)/\sqrt{2}=\sqrt{2}\) and \(w_2^\top x=(-3-1)/\sqrt{2}=-2\sqrt{2}\). Reconstructing from both axes gives
The score coordinates changed from \((3,-1)\) to \((\sqrt{2},-2\sqrt{2})\), while the full tied-block reconstruction is unchanged. If only one of these two rotated axes were retained, its projection would generally differ from the projection onto \(e_2\) alone; including the whole block is what guarantees invariance.
Interpreting Nonuniqueness Correctly
A repeated eigenvalue does not make PCA unusable. It changes what can be interpreted uniquely. The eigenspace and its projection are well-defined, while a particular basis within the space is a coordinate choice. When only part of the tied block is retained, even the selected subspace can be nonunique, although all choices attain the same variance objective.
This distinction matters when comparing software output, reporting loadings, or interpreting component scores. Two analyses can give different-looking axes for a repeated block without disagreeing about the subspace. Conversely, if truncation cuts through the block, different selected axes can yield different observation-level reconstructions; reporting only that the solutions have equal PCA error would not mean that their reconstructed data are identical.
A useful way to check a claimed invariance is to compare projection matrices. For two orthonormal bases \(V\) and \(W\), equality \(VV^\top=WW^\top\) means that they span the same subspace and produce the same projection reconstruction for every \(x\). If the matrices differ, the reconstructions can differ for some observations, even when both subspaces are optimal under the same tied-eigenvalue criterion.
Check Your Understanding
Use the distinction between a basis, its eigenspace, and its projection to answer the following.
- If \(V\) spans an eigenspace with eigenvalue \(\lambda\) and \(Q\) is orthogonal, what are the eigenvalue and projection matrix associated with \(VQ\)?
- Why can scores change after rotating a complete repeated-eigenvalue block even though the reconstruction does not?
- For \(\Sigma=5I_2\), compare the rank-one projections of \(x=(2,1)^\top\) onto \(e_1\) and \(e_2\). Are the reconstructions equal? Are the captured variances equal?
- What feature of a truncation determines whether rotating a tied block preserves the reconstruction?
- Explain why different rank-\(K\) choices that cut through a repeated-eigenvalue block can have equal PCA error but different reconstructed observations.