Bootstrap Confidence Sets for Eigenspaces

Why a Subspace, Rather Than a List of Eigenvectors?

Principal component analysis often reports eigenvectors, but an eigenvector is not uniquely determined: if $u$ is an eigenvector, then $-u$ is the same direction. More importantly, when two eigenvalues are close, small changes in the data can rotate their eigenvectors substantially. The span of several eigenvectors can nevertheless be estimated stably. Inference should therefore focus on the subspace when the scientific object is a collection of principal directions.

A bootstrap confidence set describes the range of subspaces compatible with the observed sampling variation. It is not a separate interval for each coordinate of an eigenvector. The construction below compares subspaces through their projection matrices, so it does not mistake a sign reversal for a change in direction.

Definition: Let $\Sigma$ be a symmetric covariance matrix with ordered eigenvalues $\lambda_1\geq\cdots\geq\lambda_p$. If $\lambda_k>\lambda_{k+1}$, the leading $k$-dimensional eigenspace is uniquely defined, even if eigenvalues within the first $k$ are repeated. If $U$ is a $p\times k$ matrix whose columns are an orthonormal basis for that space, its orthogonal projection matrix is $P=UU^\mathsf{T}$. The matrix $P$ does not depend on which orthonormal basis $U$ is chosen.

For data vectors $X_1,\ldots,X_n$, let $\hat\Sigma$ be the sample covariance matrix and let $\hat P$ project onto its leading $k$-dimensional eigenspace. In an ordinary nonparametric bootstrap for independent observational units, each bootstrap sample resamples $n$ units with replacement and produces a covariance matrix $\hat\Sigma^*$ and a corresponding projection $\hat P^*$. The resampling unit must match the sampling design; for dependent or clustered data, the resampling scheme must preserve that structure.

Distances Between Eigenspaces

Let $U$ and $V$ be orthonormal basis matrices for two $k$-dimensional subspaces. The singular values of $U^\mathsf{T}V$ are $\cos\theta_1,\ldots,\cos\theta_k$, where $0\leq\theta_1\leq\cdots\leq\theta_k\leq\pi/2$. These are the principal angles between the subspaces. In particular, $\theta_k$ is the largest angle between them.

Two useful distances are the largest-angle distance $\sin\theta_k$, and the Frobenius distance between projection matrices. The first emphasizes the worst-aligned direction; the second aggregates all $k$ directions.

Theorem: For rank-$k$ orthogonal projection matrices $P=UU^\mathsf{T}$ and $Q=VV^\mathsf{T}$, with principal angles $\theta_1,\ldots,\theta_k$, $$ \|P-Q\|_F^2=2\sum_{j=1}^k\sin^2\theta_j, \qquad \|P-Q\|_{\mathrm{op}}=\sin\theta_k. $$

Proof. Since $P$ and $Q$ are symmetric and idempotent, $(P-Q)^2=P+Q-PQ-QP$. Taking traces and using $\operatorname{tr}(PQ)=\operatorname{tr}(U^\mathsf{T}VV^\mathsf{T}U)$ gives $$ \|P-Q\|_F^2 =\operatorname{tr}((P-Q)^2) =2k-2\operatorname{tr}(U^\mathsf{T}VV^\mathsf{T}U) =2k-2\sum_{j=1}^k\cos^2\theta_j =2\sum_{j=1}^k\sin^2\theta_j. $$ For the operator norm, decompose the two subspaces into their principal-vector planes. On the plane associated with $\theta_j$, the difference of the two projections has eigenvalues $\sin\theta_j$ and $-\sin\theta_j$; directions common to both subspaces contribute zero. Thus its largest singular value is $\max_j\sin\theta_j=\sin\theta_k$. This also covers $\theta_j=0$, for which the contribution is zero. $\square$

Worked Example: A Sign-Invariant Comparison

Consider two rank-one subspaces in two dimensions with unit vectors $u=(0.8,0.6)^\mathsf{T}$ and $v=(0.6,0.8)^\mathsf{T}$. Their projection matrices are $$ P=uu^\mathsf{T}= \begin{pmatrix}0.64&0.48\\0.48&0.36\end{pmatrix}, \qquad Q=vv^\mathsf{T}= \begin{pmatrix}0.36&0.48\\0.48&0.64\end{pmatrix}. $$

The inner product is $u^\mathsf{T}v=0.8(0.6)+0.6(0.8)=0.96$, so the principal angle is $\theta=\arccos(0.96)$, approximately $16.26$ degrees. Its sine is $\sqrt{1-0.96^2}=0.28$. Directly, $P-Q$ is diagonal with entries $0.28$ and $-0.28$, so $\|P-Q\|_{\mathrm{op}}=0.28$ and $\|P-Q\|_F=\sqrt{0.28^2+(-0.28)^2}\approx0.396$, as the theorem predicts.

If $v$ is replaced by $-v$, then $(-v)(-v)^\mathsf{T}=Q$: both distances remain unchanged. A coordinate-by-coordinate difference between $u$ and $-v$, by contrast, would incorrectly suggest a large change.

Constructing a Bootstrap Confidence Set

A confidence set is built from the bootstrap distribution of the distance between the resampled and observed subspaces. For example, use the operator distance $d(P,Q)=\|P-Q\|_{\mathrm{op}}$, which equals the sine of the largest principal angle. Let $c^*_{1-\alpha}$ be the conditional $1-\alpha$ quantile of $\sqrt n\,d(\hat P^*,\hat P)$ over bootstrap resamples. The corresponding set consists of all rank-$k$ orthogonal projections sufficiently close to $\hat P$.

Definition: The bootstrap confidence set for the population rank-$k$ eigenspace, represented by its projection matrix, is $$ \mathcal C_{1-\alpha} = \left\{Q:\ Q^\mathsf{T}=Q,\ Q^2=Q,\ \operatorname{tr}(Q)=k,\ \sqrt n\,\|Q-\hat P\|_{\mathrm{op}}\leq c^*_{1-\alpha}\right\}. $$ Equivalently, its largest-principal-angle radius is $\arcsin\!\left(\min\{1,c^*_{1-\alpha}/\sqrt n\}\right)$. A Frobenius-distance set can be constructed in the same way by bootstrapping $\sqrt n\,\|\hat P^*-\hat P\|_F$.

This set is a ball around the observed subspace within the space of rank-$k$ projections. It is not restricted to projections that are eigenprojections of $\hat\Sigma$: those would generally include only $\hat P$ itself. The set instead describes uncertainty about the unknown population projection.

Theorem: Suppose $X_1,\ldots,X_n$ are independent and identically distributed vectors in fixed dimension $p$, with finite fourth moments. Suppose the population covariance $\Sigma$ has $\lambda_k>\lambda_{k+1}$, and the ordinary nonparametric bootstrap consistently estimates the sampling distribution of the leading-$k$ projection. More precisely, suppose the conditional distribution of $\sqrt n(\hat P^*-\hat P)$ converges in probability to the same limit as the distribution of $\sqrt n(\hat P-P)$. If the limit distribution of $\sqrt n\|\hat P-P\|_{\mathrm{op}}$ has a continuous distribution function and a unique $1-\alpha$ quantile, then $$ P(P\in\mathcal C_{1-\alpha})\longrightarrow 1-\alpha. $$

Proof. Write $F_n(t)=P\{\sqrt n\|\hat P-P\|_{\mathrm{op}}\leq t\}$, and let $F$ be its limiting distribution function. The stated distributional convergence gives $F_n(t)\to F(t)$ at every continuity point of $F$. Conditional bootstrap consistency gives convergence in probability of the bootstrap distribution function to $F$ at those points. Because $F$ is continuous and its $1-\alpha$ quantile $c$ is unique, the bootstrap quantile $c^*_{1-\alpha}$ converges in probability to $c$. The event that the true projection is in the set is exactly $$ \{P\in\mathcal C_{1-\alpha}\} = \{\sqrt n\|\hat P-P\|_{\mathrm{op}}\leq c^*_{1-\alpha}\}. $$ The statistic on the right converges in distribution to a random variable with distribution function $F$, while the random cutoff converges in probability to $c$. Continuity of $F$ at $c$ therefore gives limiting coverage $F(c)=1-\alpha$. $\square$

The eigengap condition at the boundary, $\lambda_k>\lambda_{k+1}$, is essential: it identifies a unique population subspace and makes its projection a locally smooth function of the covariance matrix. A useful sufficient setting for the stated bootstrap consistency is fixed dimension, finite fourth moments, and this nonzero boundary eigengap. Repeated eigenvalues within the selected group do not invalidate inference for the whole group. The theorem is an asymptotic statement, not a guarantee of exact finite-sample coverage.

Worked Example: Turning Bootstrap Angles Into a Confidence Set

Suppose $n=100$, $k=1$, and the observed leading direction is represented by $\hat P$. For illustration, suppose nine bootstrap resamples produce largest-angle sines $0.04,0.06,0.07,0.08,0.10,0.11,0.13,0.16,0.20$. Use $\alpha=0.20$ and the empirical quantile defined as the smallest value whose empirical cumulative proportion is at least $1-\alpha=0.80$.

The sorted list already has the displayed order. At least $0.80(9)=7.2$ observations are needed, so the first order statistic meeting that criterion is the eighth, $0.16$. The bootstrap statistic is scaled by $\sqrt n=10$, so its quantile is $c^*_{0.80}=10(0.16)=1.6$. Dividing by $\sqrt n$ gives an operator-distance radius of $1.6/10=0.16$.

Thus the illustrative confidence set includes each rank-one projection $Q$ satisfying $\|Q-\hat P\|_{\mathrm{op}}\leq0.16$. In angle terms, it includes directions within $\arcsin(0.16)$, approximately $9.21$ degrees, of the observed direction. Nine resamples are far too few for practical inference; the calculation shows the quantile conversion, not a recommended number of resamples.

Reading Uncertainty and Diagnosing Instability

A narrow confidence set indicates that the estimated subspace changes little under resampling. A wide set indicates that the data do not determine its orientation precisely. This can arise from limited sample size, noisy covariance estimates, or a small eigengap at the boundary of the selected subspace. Reporting the set or its angle radius makes that uncertainty visible.

Individual eigenvectors require a more careful interpretation. A sign flip has no effect on the one-dimensional subspace, so it should not be described as substantive instability. A small gap between adjacent eigenvalues can, however, permit genuine rotation of their individual eigenvectors. If both directions are included in the selected subspace and there is a substantial gap between $\lambda_k$ and $\lambda_{k+1}$, the combined subspace may remain stable even while its internal eigenvectors change markedly.

Worked Example: A Small Eigengap Can Rotate an Eigenvector

Compare $A=\begin{pmatrix}2&0\\0&1.99\end{pmatrix}$ with the slightly perturbed symmetric matrix $B=\begin{pmatrix}2&0.02\\0.02&1.99\end{pmatrix}$. The leading eigenvector of $A$ is $(1,0)^\mathsf{T}$. For a matrix $\begin{pmatrix}a&b\\b&d\end{pmatrix}$, the angle $\phi$ of its leading eigenvector from the first coordinate axis satisfies $\tan(2\phi)=2b/(a-d)$ when $a>d$.

For $B$, $2b/(a-d)=0.04/0.01=4$. Hence $\phi=\tfrac12\arctan(4)$, approximately $37.98$ degrees. The perturbation changed the off-diagonal entry by only $0.02$, yet the leading direction rotated substantially because the original eigenvalue gap was just $0.01$. For comparison, if the diagonal entries were $2$ and $1$ with the same off-diagonal perturbation $0.02$, then $\tan(2\phi)=0.04$, giving $\phi\approx1.15$ degrees. The size of the gap helps explain the differing sensitivity.

A practical report should state the selected dimension $k$, the estimated boundary eigengap, the distance used, and the bootstrap angle or distance quantile. Inspecting bootstrap eigenvectors is also useful, but their signs should first be aligned with a reference direction (for example, by choosing the sign that makes their inner product with the reference nonnegative). If eigenvalues are nearly tied, even sign alignment cannot remove real rotations; compare the relevant subspaces instead.

Common Pitfall

Wrong reasoning: “Some bootstrap eigenvectors point in the opposite direction from the observed eigenvector, so the estimated principal component is highly variable.” This treats $u$ and $-u$ as different directions and can inflate a measure of variation for a purely representational sign change.

Correction: Compare projection matrices or principal angles, which are invariant to the choice of eigenvector sign. Then check the eigengap: a sign reversal alone is harmless, whereas broad variation in the subspace projection can reflect genuine uncertainty. Also distinguish a small gap within the selected group, which can destabilize individual eigenvectors, from a small gap at its boundary, which threatens identification of the selected subspace itself.

Key takeaway: Bootstrap uncertainty for a principal subspace is naturally expressed as a confidence set of rank-$k$ projection matrices. Principal-angle distances remove eigenvector sign ambiguity; a clear boundary eigengap and valid bootstrap approximation underpin the usual asymptotic coverage claim.

Check Your Understanding

Use the definitions and arguments above to answer the following.

Why does the projection matrix $uu^\mathsf{T}$ remain unchanged when a unit eigenvector $u$ is replaced by $-u$?
For two rank-$k$ subspaces, what does the largest principal angle measure, and how is it related to the operator distance between their projections?
In the bootstrap confidence set, why is the resampled projection compared with $\hat P$, rather than with the unknown population projection $P$?
Which eigengap condition identifies the leading $k$-dimensional population subspace? Do repeated eigenvalues within that subspace necessarily violate it?
How can individual eigenvectors vary greatly while the selected subspace remains relatively stable?