Eigen-Gap Diagnostics for PCA Uncertainty

Why an Eigenvalue Gap Is a Diagnostic

PCA reports ordered eigenvalues and corresponding eigenvectors of a covariance matrix. A reported direction can look precise because its sample eigenvalue is separated from its neighbors, or unstable because a small change in the data could rotate its eigenvector substantially. The relevant question is not simply whether a sample gap is positive. It is whether that gap is large relative to the sampling fluctuation in the covariance matrix.

This distinction matters because an isolated population eigenvalue has a well-defined individual eigenvector, up to sign. When eigenvalues are equal, the corresponding individual eigenvectors are not uniquely defined: only their joint eigenspace is. A small but nonzero gap falls between these cases. It can permit a simple-eigenvalue approximation in large samples while still making that approximation poor at the sample size in hand.

For concreteness, suppose \(\mathbf X_1,\ldots,\mathbf X_n\) are independent, identically distributed \(p\)-dimensional observations with mean \(\boldsymbol\mu\), covariance \(\Sigma\), and finite fourth moments. Use \(S=n^{-1}\sum_{i=1}^n(\mathbf X_i-\bar{\mathbf X})(\mathbf X_i-\bar{\mathbf X})'\). Using the divisor \(n-1\) instead does not change the first-order diagnostics below. In a fixed-dimensional setting, the entries of \(S-\Sigma\) fluctuate on the order of \(n^{-1/2}\). Consequently, a gap of the same order as those fluctuations cannot reliably distinguish two individual directions.

Two Complementary Gap Diagnostics

A first diagnostic compares an estimated gap with the estimated sampling standard deviation of that gap. For a simple population eigenvalue, the first-order change in \(\lambda_j\) under a small covariance perturbation \(E\) is \(u_j'Eu_j\), where \(u_j\) is its unit eigenvector. Thus, for adjacent simple eigenvalues, the first-order influence of their gap is the difference between their two eigenvalue influences.

This result follows from the first-order eigenvalue perturbation formula and the multivariate central limit theorem applied to the sample averages of the squared projections. Centering by \(\bar{\mathbf X}\) rather than the population mean changes the covariance estimator by a term of order \(n^{-1}\), which does not affect this \(n^{1/2}\)-scaled limit. The formula requires distinct eigenvalues for the two individual eigenvalue derivatives; it is not a simple-eigenvalue standard error at a repeated eigenvalue.

The gap’s standard error alone does not fully describe eigenvector stability. Rotation depends on the covariance perturbation’s off-diagonal entries in the population eigenbasis. For \(k\ne j\), define \(\sigma_{jk}^2=\operatorname{Var}\{(u_j'(\mathbf X-\boldsymbol\mu))(u_k'(\mathbf X-\boldsymbol\mu))\}\). When the eigenvalues are distinct and the gaps are fixed as \(n\) grows, the first-order approximation to the squared angular error of the sign-aligned sample eigenvector is \[ \operatorname{E}\{\sin^2\angle(\hat u_j,u_j)\} \ \approx\ \frac{1}{n}\sum_{k\ne j}\frac{\sigma_{jk}^2}{(\lambda_j-\lambda_k)^2}. \] The right-hand side is an approximation in the separated-eigenvalue regime, not a confidence bound that remains valid near a tie. Its sample version replaces the population quantities with estimates. A large value warns that the individual direction may be poorly determined.

Both diagnostics are useful, but they answer different questions. The gap-to-standard-error ratio asks whether the gap itself is resolved relative to its sampling variability. The angular-error approximation asks how much off-diagonal sampling noise can rotate the eigenvector, with each contribution magnified by the inverse squared separation from another eigenvalue. Neither has a universal cutoff that makes an approximation valid in every model.

Why the Eigenvector Approximation Depends on the Gap

Proof. Write \(\hat u_j=a u_j+w\), where \(w\) is orthogonal to \(u_j\), and choose the sign so that \(a\ge0\). The perturbed eigenvalue equation is \((\Sigma+E)\hat u_j=\hat\lambda_j\hat u_j\). Weyl’s eigenvalue bound gives \(|\hat\lambda_j-\lambda_j|\le\|E\|\). On the orthogonal complement of \(u_j\), the distance from \(\lambda_j\) to the other eigenvalues is at least \(g\); for sufficiently small \(\|E\|/g\), the corresponding perturbed denominators remain bounded away from zero by a fixed multiple of \(g\). Projecting the eigenvalue equation onto that complement then gives \(\|w\|=O(\|E\|/g)\). Since \(a^2+\|w\|^2=1\) and \(a\ge0\), \(a=1+O(\|E\|^2/g^2)\).

Now take the inner product of the eigenvalue equation with \(u_k\), for \(k\ne j\). It gives \((\hat\lambda_j-\lambda_k)(u_k'w)=a\,u_k'Eu_j+u_k'Ew\). Here \(u_k'Ew=O(\|E\|^2/g)\), and replacing \(\hat\lambda_j-\lambda_k\) by \(\lambda_j-\lambda_k\) in the leading term incurs an error of order \(\|E\|^2/g^2\). Thus \(u_k'w=(u_k'Eu_j)/(\lambda_j-\lambda_k)+O(\|E\|^2/g^2)\). Combining these coordinates, and using the bound on \(a-1\), proves the expansion. \(\square\)

For the sample covariance, the off-diagonal entry \(u_k'(S-\Sigma)u_j\) has first-order fluctuation equal to the sample average of \((u_k'(\mathbf X_i-\boldsymbol\mu))(u_j'(\mathbf X_i-\boldsymbol\mu))\). Its variance is \(\sigma_{jk}^2/n\). Substituting these entries into the expansion yields the angular-error approximation above. The inverse gap in the expansion explains why a direction can be sensitive even when the covariance estimator itself has modest sampling error.

Worked Diagnostics

Interpreting Near-Multiplicity

A useful practical assessment combines the gap estimates, their sampling variability, and the eigenvector rotation scale. A bootstrap or an analytic influence calculation can help estimate variability, but a procedure that assumes a simple eigenvalue should not be treated as reliable evidence when the data suggest a near tie. In particular, individual eigenvectors can change sharply under small perturbations even while the subspace spanned by a cluster remains stable.

• Well-separated gap: If the estimated gap is large relative to its estimated standard error and the angular-error diagnostic is small, a simple-eigenvalue approximation is more plausible. This is evidence to assess, not a guarantee.
• Marginal gap: If the gap is comparable to its estimated sampling variability, report that the ordering and individual direction may be sensitive to sampling.
• Near-multiplicity: If several adjacent eigenvalues are difficult to distinguish, focus interpretation on the corresponding eigenspace rather than assigning substantive meaning to each basis vector separately.

The threshold for “large” depends on the purpose of the analysis and the consequences of directional error. A direction used for exploratory description may tolerate more uncertainty than one used as a fixed input to a downstream decision. Report the estimated gap and its uncertainty, state the approximation being used, and avoid presenting a numerical ratio as a universal pass/fail rule.

Common Pitfall

The key diagnostic is therefore not whether \(\hat g_j\) is positive, but whether the observed separation is meaningfully larger than the perturbations that the sample size and data distribution can produce. When that comparison is unfavorable, near-multiplicity is a substantive feature of the uncertainty, not a numerical inconvenience.