Bootstrap Confidence Regions for Eigenspaces

Why Resample an Eigenspace?

A sample principal component is estimated from a covariance matrix that would change if the observations changed. The resulting uncertainty concerns the span of the leading components, not the particular signed eigenvectors used to represent that span. A bootstrap can approximate this uncertainty by repeatedly resampling observations, recomputing the covariance matrix, and measuring how far its leading eigenspace moves.

The central object is therefore a confidence region for a subspace. To avoid arbitrary signs and rotations within the subspace, represent it by an orthogonal projector. The construction below uses the nonparametric bootstrap and is justified when the population eigenspace is separated from the remaining eigenspaces by a positive eigenvalue gap.

Projectors and Distances Between Eigenspaces

The empirical covariance matrix \(\hat{\Sigma}\) has a corresponding leading projector \(\hat{P}_r\). A bootstrap covariance matrix \(\hat{\Sigma}^{*}\) gives a bootstrap projector \(\hat{P}_r^{*}\). These are rank-\(r\) symmetric matrices satisfying \(P^2=P\) and \(\operatorname{tr}(P)=r\).

For two rank-\(r\) subspaces, let \(\theta_1,\ldots,\theta_r\) be their principal angles, each in \([0,\pi/2]\). The angles describe how the subspaces are oriented relative to one another. A useful aggregate distance is \(d_F(P,Q)=\|P-Q\|_F/\sqrt{2}\), where \(\|A\|_F=\{\operatorname{tr}(A^\mathsf{T}A)\}^{1/2}\).

To verify the identity, expand the squared Frobenius norm: \(\|P-Q\|_F^2=\operatorname{tr}\{(P-Q)^2\}=2r-2\operatorname{tr}(PQ)\), using \(P^2=P\), \(Q^2=Q\), and \(\operatorname{tr}(P)=\operatorname{tr}(Q)=r\). The singular values of \(U^\mathsf{T}V\), for orthonormal bases \(U,V\) of the two spaces, are \(\cos(\theta_j)\); consequently \(\operatorname{tr}(PQ)=\sum_j\cos^2(\theta_j)\). Substitution gives the identity. \(\square\)

Bootstrap Construction

Suppose the observed data are independent vectors \(X_1,\ldots,X_n\in\mathbb{R}^p\), drawn from a common distribution with finite fourth moments and covariance matrix \(\Sigma\). For the nonparametric bootstrap, draw \(n\) vectors with replacement from the observed vectors. Recenter each bootstrap sample when computing its covariance, and use the same covariance convention (divisor \(n\), or \(n-1\)) for every replicate and for the original sample. Recompute the leading rank-\(r\) projector for each replicate.

The resulting confidence region is a set of rank-\(r\) orthogonal projectors:

This is a region around the observed subspace in the geometry of projectors. It is not a collection of separate intervals for eigenvector coordinates. In computation, the quantile is estimated from \(B\) replicates; its accuracy depends on \(B\), so a large number of replicates is often needed when precise tail calibration matters.

Why the Bootstrap Region Has Asymptotic Coverage

A key condition is an eigengap at the boundary of the leading subspace: \(\lambda_r>\lambda_{r+1}\). Eigenvalues within the leading group may coincide; what matters for differentiability of the rank-\(r\) projector is separation of that group from the rest. Assume \(p\) and \(r\) are fixed, the observations are independent and identically distributed with finite fourth moments, and the limiting distribution of the projector distance is nondegenerate and continuous at its \((1-\alpha)\)-quantile.

Proof. The covariance matrix has a central limit theorem: \(\sqrt{n}(\hat{\Sigma}-\Sigma)\xrightarrow{d}G\), where \(G\) is a mean-zero Gaussian symmetric matrix determined by the fourth moments of the observations. The nonparametric bootstrap consistently estimates this covariance limit conditionally in probability. In particular, the conditional law of \(\sqrt{n}(\hat{\Sigma}^{*}-\hat{\Sigma})\) approaches the law of \(G\).

It remains to transfer these limits to the projector. In an orthonormal eigenbasis of \(\Sigma\), write \(\Sigma=\operatorname{diag}(\Lambda_1,\Lambda_2)\), where \(\Lambda_1\) contains the leading \(r\) eigenvalues and \(\Lambda_2\) the rest. A nearby invariant subspace can be represented as the columns of \(\begin{bmatrix}I\\ Z\end{bmatrix}\), for a small \((p-r)\times r\) matrix \(Z\). For a symmetric perturbation \(H\), write its blocks as \(H_{11},H_{12},H_{21},H_{22}\). Invariance under \(\Sigma+H\) requires

Keeping the terms linear in \(H\) and \(Z\) gives \(H_{21}+\Lambda_2Z=Z\Lambda_1\). Entrywise, \(Z_{ji}=H_{ji}/(\lambda_i-\lambda_j)\), for \(i\le r<j\). The eigengap bounds every denominator away from zero. The omitted terms contain products of perturbation blocks and \(Z\); the gap and the local invariant-subspace representation give a remainder of order \(\|H\|^2\) for sufficiently small \(H\). Thus the projector map has linear derivative

where \(u_i\) are orthonormal population eigenvectors. The first-order change in the projector depends only on perturbations linking the leading and remaining subspaces. Applying this derivative to the sample covariance limit and to the conditional bootstrap covariance limit shows that both \(\sqrt{n}(\hat{P}_r-P_r)\) and, conditionally, \(\sqrt{n}(\hat{P}_r^{*}-\hat{P}_r)\) have the same limiting law \(\mathcal{L}_{\Sigma}(G)\). Therefore their normalized Frobenius distances converge to the same continuous distance distribution. Continuity at the target quantile ensures convergence of the bootstrap quantile; the probability that the population projector lies within the resulting radius consequently tends to \(1-\alpha\). \(\square\)

Interpretation and a Common Pitfall

The region quantifies sampling uncertainty about the leading subspace under the assumed sampling model. A large radius means the resampled leading subspaces vary substantially; a small radius means they tend to remain close to the observed subspace. The distance combines movement across all principal angles, so it does not identify which direction within a multi-dimensional subspace is most uncertain. Nor does the region assert that individual eigenvectors, with particular signs, are close.

• Pitfall: comparing signed eigenvectors directly. Two eigenvectors \(u\) and \(-u\) represent exactly the same one-dimensional subspace, but their Euclidean distance is \(2\). The correction is to compare projectors, or equivalently principal angles, so the result is sign-invariant.
• Pitfall: treating bootstrap variation as a guarantee of validity in every spectrum. The coverage argument uses a positive eigengap separating the target subspace from the rest. A bootstrap calculation can still produce numbers when that assumption is doubtful; numerical output alone does not establish reliable coverage.
• Pitfall: resampling coordinates independently. PCA depends on the joint covariance of each observed vector. Resample whole vectors, not separate coordinates, to preserve the observed cross-coordinate dependence.

A bootstrap confidence region is most useful when the inferential target is the subspace itself and a basis-invariant summary is needed. Its interpretation should always include the chosen rank \(r\), distance, resampling scheme, and the evidence that the boundary eigengap is adequate.