Eigengaps and Stability of Principal Subspaces

Why the Boundary Gap Matters

Principal component analysis selects directions associated with large eigenvalues of a covariance matrix. In practice, that covariance matrix may change: a different sample, a small measurement error, or a modest change in preprocessing can produce a different matrix and therefore different principal directions. The key question is not simply whether the eigenvalues are distinct. For a retained principal subspace, the crucial separation is between the last retained eigenvalue and the first discarded eigenvalue.

If those boundary eigenvalues are close, a small change in the covariance matrix can substantially rotate directions across the boundary. If they are well separated, a comparable change is less consequential. This tutorial develops that idea through exact calculations and a result about preservation of the eigenvalue gap. It does not give a general upper bound on subspace rotation; that requires additional perturbation analysis.

Definition: Let \(A\) be a real symmetric \(p\times p\) covariance matrix, with eigenvalues ordered as \(\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\). For a chosen dimension \(K\), the retained–discarded eigengap is \(g_K=\lambda_K-\lambda_{K+1}\), when \(K<p\). If \(g_K>0\), the subspace spanned by eigenvectors for the largest \(K\) eigenvalues is uniquely determined. The individual eigenvectors need not be unique if eigenvalues within the retained group are repeated.

A principal subspace is the span of the retained eigenvectors, not a particular ordered list of axes. This distinction matters: changes in the axes within a retained subspace do not change the subspace itself. A tie among retained eigenvalues may make those axes nonunique while leaving the retained subspace well defined. In contrast, if \(\lambda_K=\lambda_{K+1}\), the boundary cuts through a tied eigenspace, so the choice of which directions to retain is not unique.

For a perturbed covariance matrix \(B=A+E\), the matrix \(E\) is assumed symmetric, as it is when both \(A\) and \(B\) are covariance matrices. Its operator norm is \(\|E\|_2=\max_{\|x\|_2=1}\|Ex\|_2\). This norm measures the largest amount by which \(E\) can change a unit vector. The eigengap is measured in the same scale as the eigenvalues; comparing \(\|E\|_2\) with \(g_K\) is therefore meaningful.

An Exact Two-Dimensional Calculation

The simplest calculation already reveals the role of the gap. Suppose the original covariance matrix is diagonal, with eigenvectors along the coordinate axes, and the larger eigenvalue is separated from the smaller one. Add a symmetric perturbation with entries \(a,b,d\). The off-diagonal entry \(b\) couples the two original directions; the difference \(a-d\) also changes their relative eigenvalue separation.

Result: Let \(A=\begin{pmatrix}\lambda_1&0\\0&\lambda_2\end{pmatrix}\), where \(\lambda_1>\lambda_2\), and let \(E=\begin{pmatrix}a&b\\b&d\end{pmatrix}\). If \(\lambda_1+a>\lambda_2+d\), the leading eigenvector of \(A+E\) can be written as \((\cos\theta,\sin\theta)^\mathsf{T}\) with \(-\pi/4<\theta<\pi/4\), and \(\tan(2\theta)=2b/[(\lambda_1-\lambda_2)+(a-d)]\).

To verify the formula, write the perturbed matrix as \(\begin{pmatrix}u&b\\b&v\end{pmatrix}\), where \(u=\lambda_1+a\) and \(v=\lambda_2+d\). For a unit vector \((\cos\theta,\sin\theta)^\mathsf{T}\), its Rayleigh quotient is \(u\cos^2\theta+2b\sin\theta\cos\theta+v\sin^2\theta\). The leading eigenvector maximizes this quantity. Differentiating with respect to \(\theta\) and setting the derivative to zero gives \((v-u)\sin(2\theta)+2b\cos(2\theta)=0\). Since \(u>v\), the maximizing direction with \(-\pi/4<\theta<\pi/4\) satisfies \(\tan(2\theta)=2b/(u-v)\), which is the stated expression. The case \(b=0\) gives \(\theta=0\), as expected. \(\square\)

When the diagonal entries are unchanged, \(a=d=0\), the angle is \(\theta=\tfrac12\arctan(2b/g)\), where \(g=\lambda_1-\lambda_2\). For a fixed small \(b\), this angle is approximately \(b/g\). Thus the same off-diagonal perturbation can have very different effects depending on the gap. This approximation is an interpretation of the exact formula, not a substitute for it when the perturbation is large.

Worked Example: The Same Perturbation, Two Different Gaps

Compare two covariance matrices, each perturbed by the same off-diagonal amount \(b=0.2\): \(A_1=\begin{pmatrix}5&0\\0&3\end{pmatrix}\) and \(A_2=\begin{pmatrix}2.1&0\\0&1.9\end{pmatrix}\). In both cases let \(E=\begin{pmatrix}0&0.2\\0.2&0\end{pmatrix}\). The first gap is \(5-3=2\), and the second is \(2.1-1.9=0.2\).

For \(A_1\), the exact formula gives \(\theta_1=\tfrac12\arctan(0.4/2)=\tfrac12\arctan(0.2)\approx0.0987\) radians, or about \(5.65^\circ\). For \(A_2\), it gives \(\theta_2=\tfrac12\arctan(0.4/0.2)=\tfrac12\arctan(2)\approx0.5536\) radians, or about \(31.72^\circ\).

The perturbation is identical, but the second leading direction rotates much more because its original gap is smaller. Both perturbed matrices remain positive definite: their eigenvalues are \(4\pm\sqrt{1+0.04}\) for the first matrix and \(2\pm\sqrt{0.01+0.04}\) for the second, all of which are positive. Thus the comparison is between valid covariance matrices, not an artifact of leaving the covariance setting.

Stability of the Retained Subspace

For a general \(K\)-dimensional principal subspace, the corresponding geometric measure of change is the collection of principal angles between the old and new subspaces. If \(U\) and \(\widetilde U\) are matrices whose columns are orthonormal bases for the two subspaces, the cosines of those angles are the singular values of \(U^\mathsf{T}\widetilde U\). The angles do not depend on which orthonormal bases are chosen. In particular, a change of basis within the same subspace does not count as subspace movement.

The boundary eigengap has two related roles. First, it determines whether the retained subspace is uniquely specified at all. Second, when it is positive, it gives the scale against which covariance perturbations should be judged. The following result establishes that a perturbation small relative to the gap cannot close the boundary separation. It does not, by itself, quantify how many degrees the subspace rotates.

Theorem: Let \(A\) and \(B=A+E\) be real symmetric \(p\times p\) matrices, and order their eigenvalues as \(\lambda_1(A)\geq\cdots\geq\lambda_p(A)\) and \(\lambda_1(B)\geq\cdots\geq\lambda_p(B)\). For each \(j\), \(|\lambda_j(B)-\lambda_j(A)|\leq\|E\|_2\). Consequently, if \(g_K=\lambda_K(A)-\lambda_{K+1}(A)>0\), then \(\lambda_K(B)-\lambda_{K+1}(B)\geq g_K-2\|E\|_2\). In particular, the boundary remains separated if \(\|E\|_2<g_K/2\).

Proof. For a real symmetric matrix \(M\), the Courant–Fischer characterization states that \(\lambda_j(M)=\max_{\dim S=j}\min_{x\in S,\ \|x\|_2=1}x^\mathsf{T}Mx\), where the maximum is over \(j\)-dimensional subspaces \(S\). For every unit vector \(x\), the Cauchy–Schwarz inequality and the definition of operator norm give \(|x^\mathsf{T}Ex|\leq\|x\|_2\|Ex\|_2\leq\|E\|_2\). Therefore, for every unit vector \(x\), \(x^\mathsf{T}Ax-\|E\|_2\leq x^\mathsf{T}Bx\leq x^\mathsf{T}Ax+\|E\|_2\). Taking the minimum over unit vectors in any fixed subspace preserves both inequalities, and then taking the maximum over subspaces also preserves them. The characterization applied to \(A\) and \(B\) yields \(\lambda_j(A)-\|E\|_2\leq\lambda_j(B)\leq\lambda_j(A)+\|E\|_2\). This proves the eigenvalue claim. Applying its lower bound at \(j=K\) and its upper bound at \(j=K+1\) gives \(\lambda_K(B)-\lambda_{K+1}(B)\geq[\lambda_K(A)-\|E\|_2]-[\lambda_{K+1}(A)+\|E\|_2]=g_K-2\|E\|_2\). If \(\|E\|_2<g_K/2\), the right side is positive, so the perturbed boundary is separated. \(\square\)

Worked Example: A Repeated Eigenvalue Away From the Boundary

Suppose a covariance matrix has eigenvalues \(8,5,5,1\), and retain \(K=1\) direction. The boundary gap is \(8-5=3\), so the leading eigendirection is uniquely determined even though two discarded eigenvalues are equal. The repeated eigenvalue affects the choice of axes within the discarded eigenspace, not the leading direction.

Now retain \(K=2\). The boundary gap is \(5-5=0\). The eigenspace for eigenvalue \(5\) has dimension two, but only one direction from that eigenspace would be included in a two-dimensional retained subspace. There is no unique choice for that direction: any unit vector in the tied eigenspace could be selected. The top-two subspace is therefore not uniquely specified. This example shows why a repeated eigenvalue is consequential only when it occurs across the retention boundary.

How Much Perturbation Produces a Chosen Rotation?

The dependence on the gap can also be read in reverse: how large a matrix change is needed to rotate a principal direction by a prescribed angle? The next construction gives an exact answer in two dimensions. It is not a general bound for arbitrary perturbations; it is a concrete way to see the scale of the effect.

Worked Example: An Exact Rotation of Thirty Degrees

Let \(A=\begin{pmatrix}6&0\\0&2\end{pmatrix}\), whose leading direction is \(e_1=(1,0)^\mathsf{T}\) and whose gap is \(4\). Rotate this direction by \(\theta=30^\circ\), using \(R=\begin{pmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{pmatrix}\), and set \(B=RAR^\mathsf{T}\). Because \(B\) is an orthogonal rotation of \(A\), its leading direction is \(Re_1=(\sqrt{3}/2,1/2)^\mathsf{T}\).

Using \(\cos^2\theta=3/4\), \(\sin^2\theta=1/4\), and \(\sin\theta\cos\theta=\sqrt{3}/4\), multiplication gives \(B=\begin{pmatrix}5&\sqrt{3}\\\sqrt{3}&3\end{pmatrix}\). Thus \(E=B-A=\begin{pmatrix}-1&\sqrt{3}\\\sqrt{3}&1\end{pmatrix}\). Its characteristic equation is \(\det(E-\mu I)=\mu^2-4=0\), so its eigenvalues are \(2\) and \(-2\), and \(\|E\|_2=2\). A thirty-degree rotation has therefore been produced by a perturbation whose norm is half the original gap.

More generally, for a diagonal two-dimensional matrix with gap \(g>0\), rotating its eigenvectors by \(\theta\) while keeping its eigenvalues fixed gives a perturbation with operator norm \(g|\sin\theta|\). The calculation above verifies this at \(g=4\) and \(\theta=30^\circ\): \(4\sin(30^\circ)=2\). For a fixed nonzero rotation, a larger gap requires a larger perturbation in this construction.

Using the Gap in Practice

An eigengap is a diagnostic, not a guarantee that a particular sample PCA will be stable. Its interpretation depends on the scale and structure of the covariance change. A perturbation with a large operator norm may act mainly within the retained subspace and have little effect on its orientation; another with a smaller norm may couple retained and discarded directions directly. The exact two-dimensional calculation illustrates that off-diagonal coupling, relative to the gap, drives rotation in that setting. For general matrices, a quantitative angle guarantee requires a subspace perturbation result.

When comparing PCA solutions, first specify the dimension \(K\), then inspect \(\lambda_K-\lambda_{K+1}\), not merely whether the leading eigenvalues are distinct. A small boundary gap signals that the chosen truncation may be sensitive. Also compare perturbation size on the same scale as the eigenvalues: changes in measurement units or variable scaling change covariance magnitudes and can alter the practical meaning of a numerical gap.

A further distinction is useful when interpreting results. If the retained eigenvalues are close to one another but the boundary gap is large, individual component axes may be difficult to distinguish within the retained subspace, while the subspace itself can still be well determined. Conversely, a small boundary gap can make the subspace unstable even if the retained eigenvalues are distinct from each other. The object of interest—axis or subspace—must be clear before making a stability claim.

Common Mistake

Wrong reasoning: “The leading eigenvalue is distinct, so the leading principal direction must be stable.” A distinct eigenvalue establishes uniqueness, but it does not establish robustness. If its gap from the next eigenvalue is very small, a modest perturbation can rotate the direction substantially.
Correction: Compare the perturbation scale with the relevant retained–discarded gap. Distinctness gives a positive gap; stability depends on how large that gap is relative to the covariance change.
Wrong reasoning: “There is a repeated eigenvalue, so the retained subspace is necessarily nonunique.”
Correction: Check where the tie occurs. A repeated eigenvalue entirely within the retained group or entirely within the discarded group does not by itself make the retained subspace nonunique. A tie at the boundary does.
Wrong reasoning: “A positive gap proves that the subspace barely moves under any perturbation.”
Correction: A positive gap specifies uniqueness and a scale for judging sensitivity; it does not bound subspace rotation without conditions on the perturbation. Use an appropriate perturbation bound when a quantitative angle guarantee is needed.

Key takeaway: For a \(K\)-dimensional principal subspace, the gap \(\lambda_K-\lambda_{K+1}\) is the relevant separation. A positive boundary gap makes the subspace unique; a small gap warns that modest covariance changes may matter. Internal eigenvalue ties and the boundary gap answer different questions.

Check Your Understanding

Use the boundary gap and the perturbation scale to answer each question.

A covariance matrix has ordered eigenvalues \(9,6,6,2\). What is the retained–discarded gap for \(K=1\), and is the leading direction unique?
For the same eigenvalues, what is the gap for \(K=2\)? Explain why a zero gap affects uniqueness of the retained subspace.
In the two-dimensional formula with unchanged diagonal entries, what happens to the rotation angle when the off-diagonal perturbation stays fixed while the gap increases?
If \(g_K=1.2\) and \(\|E\|_2=0.4\), what lower bound does the theorem give for the perturbed boundary gap? Does it guarantee positive separation?
Why does a positive boundary gap not, by itself, give a numerical upper bound on the angle between the original and perturbed subspaces?