Eigenspace Consistency from Perturbation Bounds

Why the Eigengap Matters

PCA estimates directions of high variance from a covariance matrix. In applications with many variables, the estimated covariance matrix may differ from its population counterpart, so the estimated principal directions need not coincide with the population directions. The important question is not only how large that covariance error is, but also how it compares with the eigengap separating the subspace of interest from the rest of the spectrum.

A substantial gap makes a subspace stable to moderate perturbations. If the gap is small, even a small change in the covariance matrix can substantially rotate its eigenvectors. The perturbation bound below makes this comparison precise and gives a direct route to consistency when both dimension and rank can vary.

Subspace Distance and Covariance Perturbation

For a symmetric population covariance matrix \(\Sigma\), order its eigenvalues as \(\lambda_1\geq\cdots\geq\lambda_p\). Let \(U\) contain orthonormal eigenvectors for the leading \(r\) eigenvalues, and define the population eigengap at the boundary of that subspace by \(\delta=\lambda_r-\lambda_{r+1}\). We require \(r<p\) and \(\delta>0\), so the population leading subspace is uniquely specified. Write the estimated covariance matrix as \(\widehat\Sigma=\Sigma+E\), where \(E\) is symmetric, and let \(\widehat U\) span a leading \(r\)-dimensional eigenspace of \(\widehat\Sigma\). The perturbation size is \(\varepsilon=\|E\|_2\).

The denominator expresses the effective separation between the estimated leading eigenvalues and the population eigenvalues outside the target subspace. The bound is an upper bound, not a prediction that the actual rotation must be this large. Its value is that it applies without requiring a special covariance structure.

Proof of the Perturbation Bound

First, Weyl’s eigenvalue inequality gives \(\widehat\lambda_r\geq\lambda_r-\varepsilon\), where \(\widehat\lambda_r\) is the \(r\)th largest eigenvalue of \(\widehat\Sigma\). This inequality follows from the variational characterization of ordered eigenvalues: for every unit vector \(x\), \(|x^\top E x|\leq\|E\|_2=\varepsilon\), so each variational minimum and maximum changes by at most \(\varepsilon\). Thus every eigenvalue in the estimated leading \(r\)-dimensional subspace is at least \(\lambda_r-\varepsilon\).

Choose \(U_\perp\) as an orthonormal eigenbasis for the remaining eigenvectors of \(\Sigma\), and write \(\Lambda_\perp=U_\perp^\top\Sigma U_\perp\). Its largest eigenvalue is \(\lambda_{r+1}\). Let \(\widehat\Lambda\) be the diagonal matrix of eigenvalues corresponding to the columns of \(\widehat U\), and set \(X=U_\perp^\top\widehat U\). Since \(\widehat\Sigma\widehat U=\widehat U\widehat\Lambda\), multiplication by \(U_\perp^\top\) gives \(\Lambda_\perp X+U_\perp^\top E\widehat U=X\widehat\Lambda\), or \(X\widehat\Lambda-\Lambda_\perp X=U_\perp^\top E\widehat U\).

All eigenvalues of \(\widehat\Lambda\) are at least \(c=\lambda_r-\varepsilon\), while all eigenvalues of \(\Lambda_\perp\) are at most \(d=\lambda_{r+1}\). The separation is \(c-d=\delta-\varepsilon>0\). For symmetric matrices \(C\) and \(D\) with eigenvalues respectively at least \(c\) and at most \(d<c\), the solution \(Y\) of \(YC-DY=F\) can be written \(Y=\int_0^\infty e^{tD}F e^{-tC}\,dt\). Indeed, differentiating the integrand and integrating from zero to infinity verifies the equation; the integral converges because \(\|e^{tD}F e^{-tC}\|_2\leq e^{-(c-d)t}\|F\|_2\). Consequently, \(\|Y\|_2\leq\|F\|_2/(c-d)\). Apply this identity with \(Y=X\), \(C=\widehat\Lambda\), \(D=\Lambda_\perp\), and \(F=U_\perp^\top E\widehat U\). Since \(U_\perp\) and \(\widehat U\) have orthonormal columns, \(\|F\|_2\leq\varepsilon\). Therefore \(\|X\|_2\leq\varepsilon/(\delta-\varepsilon)\). The definition of principal angles gives \(\|X\|_2=\|\sin\Theta\|_2\), proving the bound. If \(\varepsilon\leq\delta/2\), then \(\delta-\varepsilon\geq\delta/2\), giving the stated simpler bound. \(\square\)

Consistency When Dimension Grows

Now consider a sequence of problems indexed by \(n\), with dimension \(p_n\), target rank \(r_n<p_n\), population covariance \(\Sigma_n\), and estimate \(\widehat\Sigma_n\). Let \(\delta_n=\lambda_{r_n}(\Sigma_n)-\lambda_{r_n+1}(\Sigma_n)>0\) and \(\varepsilon_n=\|\widehat\Sigma_n-\Sigma_n\|_2\). The theorem gives a direct sufficient condition for consistency of the estimated subspace in largest-principal-angle distance.

To see this, the condition implies that the event \(\varepsilon_n<\delta_n/2\) has probability tending to one. On that event the theorem gives \(\|\sin\Theta_n\|_2\leq2\varepsilon_n/\delta_n\). The right-hand side converges to zero in probability, and the event on which this bound applies has probability tending to one. Hence the subspace error converges to zero in probability.

There is also a useful distinction between operator-norm and aggregate subspace error. Let \(P=UU^\top\) and \(\widehat P=\widehat U\widehat U^\top\) be the orthogonal projection matrices. Because \(P\) and \(\widehat P\) are rank-\(r\) orthogonal projections, \(\|P-\widehat P\|_F^2=\operatorname{tr}(P)+\operatorname{tr}(\widehat P)-2\operatorname{tr}(P\widehat P)\). Here both first traces equal \(r\), and \(\operatorname{tr}(P\widehat P)=\|U^\top\widehat U\|_F^2=\sum_{j=1}^r\cos^2\theta_j\). It follows that \[ \|P-\widehat P\|_F^2 =2\sum_{j=1}^r\sin^2\theta_j =2\|\sin\Theta\|_F^2 \leq 2r\|\sin\Theta\|_2^2. \] Thus operator-norm consistency controls the largest angle even when \(r\) grows, but Frobenius consistency also requires that the accumulated error across directions vanish. A sufficient condition is \(\sqrt{r_n}\,\varepsilon_n/\delta_n\xrightarrow{p}0\).

When a Vanishing Error Is Not Enough

The covariance perturbation can converge to zero while the estimated subspace remains inconsistent if the eigengap shrinks at the same rate or faster. The condition concerns the ratio \(\varepsilon_n/\delta_n\), not \(\varepsilon_n\) in isolation.

Common Pitfall and Practical Interpretation

A common mistake is to argue, “The estimated covariance converges to the population covariance, so its principal subspace must converge.” This skips the eigengap. The correction is to compare covariance error with the population gap: a useful sufficient condition is \(\|\widehat\Sigma_n-\Sigma_n\|_2/\delta_n\xrightarrow{p}0\). If that ratio does not vanish, the Davis–Kahan bound does not guarantee consistency, and the shrinking-gap example shows that inconsistency can actually occur.

The bound is a stability statement, not a covariance-estimation method and not a rule for choosing the number of factors. Its role is to translate a bound on covariance error into a bound on the subspace error. In a high-dimensional analysis, the key inputs are therefore an operator-norm error bound and a positive separation between the target eigenvalues and the remaining spectrum.