Bootstrap Failure Near Repeated Eigenvalues

Why a Bootstrap Interval Can Look Precise and Still Mislead

A principal direction is often reported as a vector, and a bootstrap interval may summarize how that vector changes across resamples. But a direction is only a uniquely defined population target when its eigenvalue is separated from the others. If two population eigenvalues coincide, every unit vector in their shared eigenspace is an eigenvector. There is no distinguished direction for a confidence interval to cover.

The difficulty is not just the familiar sign ambiguity \(u\) versus \(-u\). Sign-invariant methods address that ambiguity when an eigenvector is identifiable. At a repeated eigenvalue, an entire rotation of directions can be equally valid. The sample may select one orientation from random perturbations, and resampling can reproduce those perturbations in a way that does not approximate a fixed population direction.

Definition: Let \(\Sigma\) be a population covariance matrix, with ordered eigenvalues \(\lambda_1 \ge \cdots \ge \lambda_p\). A simple eigenvalue has multiplicity one. Its unit eigenvector is identifiable up to sign. An eigenvalue with multiplicity greater than one has an eigenspace of dimension greater than one; no particular orthonormal basis within that eigenspace is identified by \(\Sigma\). A spectral subspace is the span of the eigenvectors associated with a specified group of eigenvalues. Its orthogonal projector does not depend on the choice of basis within that subspace.

Let \(Z_1,\ldots,Z_n\) be independent, identically distributed \(p\)-dimensional observations with finite fourth moments and covariance \(\Sigma\). Write \(S_n\) for their sample covariance matrix and \(S_n^*\) for the covariance matrix of a nonparametric bootstrap resample of the rows. The ordinary eigenvector bootstrap computes eigenvectors of \(S_n^*\), often aligning their signs with the corresponding eigenvectors of \(S_n\), and uses their variation to form intervals or regions.

When a population eigenvalue is simple and has a nonzero gap from the rest of the spectrum, the corresponding direction changes smoothly under small covariance perturbations. At a tie, that smoothness fails: perturbations of size \(n^{-1/2}\) can select a direction that differs by an angle of order one. Nearly tied eigenvalues create a related problem when their gap is on the same scale as sampling fluctuations.

What Happens at an Exact Tie

Consider a two-dimensional population covariance \(\Sigma=I_2\), where \(I_2\) is the identity matrix. The two population eigenvalues are both 1, so there is no unique leading population direction. For Gaussian observations, the centered and scaled sample covariance has an asymptotically Gaussian perturbation. After removing its trace, write that perturbation as \(\begin{pmatrix} a & b \\ b & -a \end{pmatrix}\). Its leading eigenvector has angle \(\phi=\tfrac12\operatorname{atan2}(b,a)\), considered modulo \(\pi\). The orientation is determined by the random perturbation, not by a population direction.

The same feature affects the bootstrap. Conditional on the data, the bootstrap covariance adds another fluctuation to the observed perturbation. In the Gaussian limit, if \(G=(a,b)\) denotes the original trace-free perturbation and \(G'\) an independent bootstrap-limit perturbation, the bootstrap direction is determined by \(G+G'\), while the sample direction is determined by \(G\). Thus the bootstrap's rotation relative to the sample depends on the realized \(G\). In particular, the conditional distribution of that rotation is not a fixed distribution that consistently estimates a population-direction sampling law.

Key result: At a repeated population eigenvalue, an individual eigenvector is not an identified population parameter. Even after sign alignment, ordinary bootstrap eigenvector intervals cannot be interpreted as confidence intervals for a unique population direction. In standard tied models, the conditional bootstrap distribution of the direction relative to the sample direction also has a nonvanishing random limit, rather than consistently estimating a fixed sampling distribution.

Worked Example: A Tied Gaussian Covariance

Let \(Z_i \sim N(0,I_2)\). The population covariance is \(\Sigma=\begin{pmatrix}1&0\\0&1\end{pmatrix}\), with a repeated eigenvalue. For the sample covariance \(S_n\), consider the trace-free coordinates \(A_n=\sqrt{n}(S_{n,11}-S_{n,22})/2\) and \(B_n=\sqrt{n}S_{n,12}\).

For a standard bivariate normal observation \((X,Y)\), the corresponding unscaled contributions are \((X^2-Y^2)/2\) and \(XY\). Their means are zero. Since \(E[X^2]=E[Y^2]=1\), \(E[X^4]=E[Y^4]=3\), and \(E[X^2Y^2]=1\), \(\operatorname{Var}((X^2-Y^2)/2)=(3+3-2)/4=1\) and \(\operatorname{Var}(XY)=1\). Their covariance is zero because the relevant odd moments vanish. The multivariate central limit theorem therefore gives \((A_n,B_n)\xrightarrow{d}N(0,I_2)\).

The leading sample eigenvector has angle \(\phi_n=\tfrac12\operatorname{atan2}(B_n,A_n)\), modulo \(\pi\). A rotationally symmetric bivariate normal vector has a uniform polar angle, so \(\phi_n\) has an asymptotically uniform orientation modulo \(\pi\). It does not concentrate around any particular population direction: none is specified by \(\Sigma\).

For the bootstrap, the analogous limiting trace-free coordinates are \(G+G'\), where \(G\) is the realized sample-limit vector and \(G'\) is an independent \(N(0,I_2)\) increment. If \(G\) is close to zero, the bootstrap direction is nearly uniform relative to the sample direction. If \(\lVert G\rVert\) is large, adding \(G'\) usually changes the direction only slightly. Since \(\lVert G\rVert\) remains random and is not forced toward either case, the conditional bootstrap law varies with the observed data. A sign convention cannot remove this rotational uncertainty.

Why a Small Gap Is Also a Warning

For a simple eigenvalue, the first-order change in its eigenvector contains denominators equal to eigenvalue gaps. The smaller a gap is, the larger the response to a given covariance perturbation can be. This is why an eigenvector bootstrap that behaves adequately at a comfortably separated eigenvalue can become erratic as a gap shrinks.

First-order eigenvector perturbation: Suppose \(\lambda_k\) is a simple eigenvalue of \(\Sigma\), with unit eigenvector \(u_k\), and let \(H\) be a small symmetric perturbation. With a local sign choice satisfying \(u_k^T\tilde u_k \ge 0\), the first-order change in the direction is \(\displaystyle \tilde u_k-u_k \ \approx\ \sum_{j\ne k}u_j\frac{u_j^THu_k}{\lambda_k-\lambda_j}\). The approximation requires a sufficiently small perturbation relative to the gaps; it is not a uniform expansion as a gap tends to zero.

Worked Example: A Gap Comparable to the Perturbation

Consider the population matrix \(\Sigma_\delta=\begin{pmatrix}1+\delta/2&0\\0&1-\delta/2\end{pmatrix}\), whose leading eigenvector is \(u_1=(1,0)^T\) when \(\delta>0\). Add an off-diagonal perturbation \(H=\begin{pmatrix}0&\varepsilon\\\varepsilon&0\end{pmatrix}\). The perturbed matrix has diagonal difference \(\delta\) and off-diagonal entry \(\varepsilon\).

For a symmetric \(2\)-by-\(2\) matrix with diagonal difference \(d\) and off-diagonal entry \(c\), the leading direction angle \(\theta\) satisfies \(\tan(2\theta)=2c/d\). Here this gives \(\theta=\tfrac12\arctan(2\varepsilon/\delta)\), taking the angle near zero when \(\varepsilon\) is small relative to \(\delta\).

If \(\delta=0.20\) and \(\varepsilon=0.01\), then \(\theta=\tfrac12\arctan(0.10)\), approximately \(2.86\) degrees. If instead \(\delta=0.02\) with the same perturbation, then \(\theta=\tfrac12\arctan(1)\), exactly \(22.5\) degrees. The perturbation did not change; reducing the gap by a factor of ten greatly increased the rotation. If \(\delta\) and \(\varepsilon\) are both of order \(n^{-1/2}\), the angle need not vanish as \(n\) grows. A fixed-gap approximation then gives no uniform assurance that a direction interval is reliable.

This distinction matters in finite samples. A nonzero sample gap does not establish that the corresponding population eigenvalue is well separated: sample eigenvalues split even when the population values coincide. When the gap is small relative to the scale of covariance estimation, an apparently narrow direction interval may reflect the resampling and alignment procedure rather than stable information about a population direction.

A Separated Eigenspace Is a Better Target

A repeated eigenvalue does not make every PCA target unstable. Suppose a cluster of eigenvalues is separated from the remaining eigenvalues by a positive gap, even if values within the cluster are equal. The span of the cluster is then identifiable and stable. Its projector is unchanged by any rotation of basis vectors within the cluster.

Theorem: Let \(\Sigma\) be a fixed symmetric covariance matrix, and let \(C\) be a specified group of its eigenvalues, possibly containing repetitions. Assume this group is separated from every eigenvalue outside it by a positive gap. Let \(P_C\) be the orthogonal projector onto its eigenspace. If the observations are iid with finite fourth moments, then the ordinary nonparametric bootstrap consistently estimates the first-order sampling distribution of the sample projector \(\hat P_C\): conditionally on the data, the distribution of \(\sqrt n(\hat P_C^*-\hat P_C)\) converges in probability to the same limit as \(\sqrt n(\hat P_C-P_C)\). Here the bootstrap cluster is the set of sample eigenvalues corresponding to \(C\), and convergence of conditional distributions is understood in the usual bounded-Lipschitz sense.

To see the regularity behind this result, choose an orthonormal eigenbasis \(u_1,\ldots,u_p\) of \(\Sigma\), with eigenvalues \(\lambda_1,\ldots,\lambda_p\), and write \(H\) for a small symmetric perturbation. For the projector onto the cluster \(C\), its first derivative is \(\displaystyle DP_C[H]=\sum_{i\in C,\ j\notin C}\frac{u_j^THu_i}{\lambda_i-\lambda_j}(u_ju_i^T+u_iu_j^T)\). The denominators compare cluster eigenvalues only with eigenvalues outside the cluster. No denominator involves a gap between two eigenvalues inside \(C\), so repetitions within the cluster do not make this derivative singular.

Linearize the projector.
The positive gap separating \(C\) from its complement makes the projector a differentiable function of the covariance matrix in a neighborhood of \(\Sigma\), with the derivative displayed above.

Apply covariance asymptotics.
Under iid sampling and finite fourth moments, \(\sqrt n(S_n-\Sigma)\) has a Gaussian limit. The derivative maps that limit to the first-order limit of \(\sqrt n(\hat P_C-P_C)\).

Apply the bootstrap version.
The nonparametric bootstrap consistently estimates the covariance-matrix limit under these conditions. Differentiability gives the same derivative applied to the bootstrap covariance fluctuation around \(S_n\), yielding the stated conditional approximation for the projector.

This proof uses the multivariate central limit theorem, bootstrap consistency for the sample covariance under finite fourth moments, and the differentiable mapping principle. The essential feature for this topic is that the projector's derivative remains well-defined when eigenvalues inside \(C\) coincide, as long as the boundary gap remains positive. The theorem does not assert validity for individual eigenvectors inside the cluster.

Worked Example: A Stable Two-Dimensional Subspace

Let \(\Sigma=\operatorname{diag}(5,5,1)\), and target the leading two-dimensional eigenspace. Its population projector is \(P_C=\operatorname{diag}(1,1,0)\). The two leading eigenvalues are tied, so neither leading eigenvector has a unique orientation within their plane; however, the plane is separated from the third direction by a gap of \(5-1=4\).

Consider a symmetric perturbation \(H\) whose only nonzero entries are \(H_{31}=H_{13}=0.20\) and \(H_{32}=H_{23}=-0.10\). In the projector derivative, the coefficient linking \(u_1\) to \(u_3\) is \(0.20/(5-1)=0.05\), and the coefficient linking \(u_2\) to \(u_3\) is \(-0.10/(5-1)=-0.025\). Therefore \(DP_C[H]=\begin{pmatrix}0&0&0.05\\0&0&-0.025\\0.05&-0.025&0\end{pmatrix}\).

The derivative contains no division by \(5-5\): the internal tie does not destabilize the two-dimensional target. In contrast, trying to assign separate leading eigenvectors would require selecting an orientation within the tied plane. Bootstrap inference for the projector can therefore express uncertainty about the plane without pretending that its basis is determined.

Common Mistake and Practical Interpretation

A common mistake is to see that every bootstrap replicate has a leading eigenvector and conclude that the resulting interval estimates a population leading direction. Each replicate does have an eigenvector, but at a tie that vector is selected by random sample variation. Sign alignment only chooses between two representations of the same line; it cannot identify a preferred line within a multidimensional eigenspace.

A second mistake is to treat a small but nonzero observed eigenvalue gap as evidence that direction inference is safe. The relevant question is whether the population gap is large relative to the sampling fluctuation. When it is not, a simple-eigenvalue approximation may be poor even if the empirical ordering is unambiguous. Subspace inference is often a more honest target when a group of components is stable as a whole but their individual directions are not.

Key takeaway: Ordinary eigenvector bootstrap intervals require an identifiable, sufficiently separated population direction to have a clear interpretation. At a repeated eigenvalue, individual directions are not identified; near a tie, they can be highly sensitive to sampling noise. If the eigenvalue cluster is separated from the rest of the spectrum, infer about its projector or span instead.

Check Your Understanding

Use the distinction between individual directions and subspaces to answer the following.

Why does sign alignment fail to resolve the ambiguity of an eigenvector inside a repeated-eigenvalue eigenspace?
In the tied Gaussian example, what feature of the conditional bootstrap rotation shows that its limiting distribution is random?
For a simple eigenvalue, what happens to the first-order eigenvector perturbation when its gap to a neighboring eigenvalue shrinks?
Why can a cluster projector remain differentiable when eigenvalues within the cluster coincide?
In the matrix \(\operatorname{diag}(5,5,1)\), which eigentarget is identified: either individual leading direction, or the span of both leading directions? Explain.