Reconstruction Risk in High-Dimensional PCA

Why variance explained is not the same as signal recovery

Principal components are often judged by how much variance they explain. In a high-dimensional spiked model, that phrase can refer to different quantities: variance in the training sample, population variance captured by a fitted projection, or recovery of the population direction that generated the signal. These quantities need not agree. In particular, a sample eigenvalue can be large even when its eigenvector has little alignment with the population signal.

We quantify the distinction using the population reconstruction error of an independent observation projected onto the first $k$ sample principal components. The key quantity is the squared projection of the population spike direction onto the retained sample-PC subspace. It measures signal recovery directly and determines the excess reconstruction risk over an oracle that knows the population direction.

Model, projection, and losses

Definition: Consider independent observations $X_1,\ldots,X_n\in\mathbb{R}^p$ from the rank-one spiked covariance model $$ X_i\sim N(0,\Sigma),\qquad \Sigma=I_p+\beta uu^\top, $$ where $u\in\mathbb{R}^p$ is a fixed unit vector and $\beta>0$ is the spike strength. Let $S=n^{-1}\sum_{i=1}^n X_iX_i^\top$, and let $\hat v_1,\ldots,\hat v_p$ be orthonormal sample eigenvectors ordered by decreasing sample eigenvalue. For $1\leq k\leq p$, define the retained-PC projection $\hat P_k=\sum_{j=1}^k\hat v_j\hat v_j^\top$, and its population-spike overlap $q_k=u^\top\hat P_k u=\sum_{j=1}^k(u^\top\hat v_j)^2$.

The vector $X$ used to assess reconstruction is an independent test observation with covariance $\Sigma$, independent of the training sample that produced $\hat P_k$. The population reconstruction loss, conditional on the training sample, is $E[\|(I_p-\hat P_k)X\|^2\mid \hat P_k]$. The empirical variance explained, by contrast, is $\operatorname{tr}(\hat P_kS)=\sum_{j=1}^k\hat\lambda_j$, where $\hat\lambda_j$ are the ordered eigenvalues of $S$.

Definition: The population variance captured by a rank-$k$ projection $P$ is $\operatorname{tr}(P\Sigma)$. The population reconstruction loss is $\operatorname{tr}((I_p-P)\Sigma)$. The spike recovery fraction for $\hat P_k$ is $q_k=u^\top\hat P_k u$, the fraction of the unit spike direction’s squared length lying in the retained sample-PC subspace.

Exact reconstruction-risk identity

Theorem: For any fixed rank-$k$ orthogonal projection $P$, the population variance it captures and its population reconstruction loss are $$ \operatorname{tr}(P\Sigma)=k+\beta u^\top Pu,\qquad \operatorname{tr}((I_p-P)\Sigma)=p-k+\beta(1-u^\top Pu). $$ Consequently, conditional on the training sample, retaining the first $k$ sample PCs gives population reconstruction loss $p-k+\beta(1-q_k)$.

Proof. Since $P$ is an orthogonal projection of rank $k$, $\operatorname{tr}(P)=k$. Using $\Sigma=I_p+\beta uu^\top$ and $\operatorname{tr}(Puu^\top)=u^\top Pu$, $$ \operatorname{tr}(P\Sigma)=\operatorname{tr}(P)+\beta\operatorname{tr}(Puu^\top)=k+\beta u^\top Pu. $$ Also, $\operatorname{tr}(\Sigma)=p+\beta$, since $u^\top u=1$. Subtracting the captured variance from the total population variance gives $$ \operatorname{tr}((I_p-P)\Sigma)=p+\beta-k-\beta u^\top Pu =p-k+\beta(1-u^\top Pu). $$ For an independent mean-zero test vector $X$, $E[\|(I_p-P)X\|^2]=\operatorname{tr}((I_p-P)\Sigma)$, by expanding the squared norm and using $E[XX^\top]=\Sigma$. Taking $P=\hat P_k$ conditional on the training sample proves the stated identity. $\square$

Averaging over the training sample yields $$ E\!\left[\|(I_p-\hat P_k)X\|^2\right]=p-k+\beta(1-E[q_k]). $$ Thus the isotropic noise contributes $p-k$, the amount left outside any rank-$k$ projection, while the unrecovered spike contributes $\beta(1-E[q_k])$. The expected population variance captured is $k+\beta E[q_k]$.

The oracle benchmark and the BBP regime

An oracle that knows $u$ can choose a rank-$k$ projection containing $u$, provided $k\geq1$. Such a projection has $u^\top Pu=1$, so its reconstruction loss is $p-k$. No rank-$k$ projection can do better: for every orthogonal projection $P$, $u^\top Pu=\|Pu\|^2\leq\|u\|^2=1$. The exact risk identity therefore gives $\operatorname{tr}((I_p-P)\Sigma)\geq p-k$, with equality when $Pu=u$. The sample projection’s excess risk over this oracle is exactly $\beta(1-q_k)$.

For the high-dimensional limit, suppose $p,n\to\infty$ with $p/n\to\gamma\in(0,\infty)$, while $\beta$ is fixed. In the Gaussian rank-one spiked covariance model, the BBP eigenvector-alignment result says that the leading sample eigenvector has limiting squared overlap $$ (u^\top\hat v_1)^2\xrightarrow{p} \begin{cases} 0,&\beta\leq\sqrt{\gamma},\\[4pt] \displaystyle\frac{1-\gamma/\beta^2}{1+\gamma/\beta},&\beta>\sqrt{\gamma}. \end{cases} $$ The threshold $\beta=\sqrt{\gamma}$ separates the subcritical regime, where the top sample eigenvector does not asymptotically recover the spike direction, from the supercritical regime, where its overlap is positive.

For rank one, write $q_1=(u^\top\hat v_1)^2$. Because $0\leq q_1\leq1$, convergence in probability to a constant implies convergence of its expectation to that constant: for any $\varepsilon>0$, $E|q_1-c|\leq\varepsilon+P(|q_1-c|>\varepsilon)$, where $c$ is the limiting value, and the probability term tends to zero. The risk identity therefore gives the limiting expected rank-one reconstruction loss per the exact expression $p-1+\beta(1-E[q_1])$, with the excess over the oracle tending to $\beta(1-c)$. The same interpretation applies to a fixed number $k$ of retained PCs when their total overlap $q_k$ is evaluated; the exact risk identity itself requires no asymptotic assumptions.

Worked Example: A Supercritical Spike

Let $\beta=2$ and $\gamma=0.5$, and retain only the leading sample PC. Since $2>\sqrt{0.5}$, the spike is supercritical. Its limiting squared overlap is $$ c=\frac{1-0.5/2^2}{1+0.5/2} =\frac{1-0.125}{1.25} =\frac{0.875}{1.25}=0.7. $$

The leading sample-PC projection captures limiting population variance $1+\beta c=1+2(0.7)=2.4$. Its limiting excess reconstruction loss over the rank-one oracle is $\beta(1-c)=2(0.3)=0.6$, so its total population loss is $p-1+0.6=p-0.4$. The oracle loss is $p-1$.

The outlier sample eigenvalue has limit $(1+\beta)(1+\gamma/\beta)=3(1.25)=3.75$. This is not the same as the population variance captured by the fitted direction, which tends to $2.4$. The sample eigenvalue describes variation along a direction selected using the training data; the population variance captured evaluates that direction against $\Sigma$.

Worked Example: A Subcritical Spike

Take $\beta=1$ and $\gamma=1$. The threshold is $\sqrt{\gamma}=1$, so this spike is at the threshold. The limiting squared overlap of the leading sample eigenvector is $c=0$. Hence its limiting population variance captured is $1+\beta c=1$, and its limiting reconstruction loss is $p-1+\beta(1-c)=p$.

The rank-one oracle captures the entire spike direction and has loss $p-1$. Thus the leading sample PC has limiting excess loss $1$, equal to the full spike strength. The fact that the sample’s largest eigenvalue is near the upper spectral edge does not change this conclusion: at the threshold the top eigenvector has no positive limiting overlap with the population spike.

More retained components and signal projection

The risk identity also separates total variance captured from spike recovery when more than one component is retained. For any $k$, the population variance captured is $k+\beta q_k$, not simply the number of components and not necessarily the sum of the leading sample eigenvalues. The total population reconstruction loss is $p-k+\beta(1-q_k)$. In particular, adding a component always removes one unit of isotropic-noise reconstruction loss, while the change in spike recovery depends on the additional overlap with $u$.

This same overlap has a direct signal interpretation. Write a test observation as $X=\sqrt{\beta}Zu+\varepsilon$, where $Z\sim N(0,1)$, $\varepsilon\sim N(0,I_p)$, and $Z$ and $\varepsilon$ are independent. The population signal is $s=\sqrt{\beta}Zu$. Conditional on $\hat P_k$, $$ E[\|(I_p-\hat P_k)s\|^2\mid\hat P_k] =\beta E[Z^2]\|(I_p-\hat P_k)u\|^2 =\beta(1-q_k). $$ Here $E[Z^2]=1$ and $\|(I_p-\hat P_k)u\|^2=1-u^\top\hat P_k u$, because $\hat P_k$ is an orthogonal projection. Thus the excess test-data reconstruction loss over the oracle is precisely the expected squared signal left outside the retained subspace.

Worked Example: Retaining Two Components

Continue with $\beta=2$, $\gamma=0.5$, and suppose two leading sample PCs are retained. In the standard Gaussian model, for fixed $k=2$, the second sample eigenvector’s squared overlap with $u$ tends to zero; hence $q_2$ has the same limit $0.7$ as $q_1$. The population variance captured tends to $$ 2+2(0.7)=3.4. $$

The total population variance is $p+2$, so the rank-two reconstruction loss tends to $p+2-3.4=p-1.4$. The rank-two oracle loss is $p-2$, and the excess is $0.6$, the same unrecovered spike contribution as for one retained component. The additional component removes one more unit of isotropic noise variance but, asymptotically, contributes no additional spike recovery in this fixed-rank setting.

Common mistake: reading recovery directly from a sample eigenvalue

Wrong reasoning: “The leading sample eigenvalue is well above the bulk, so the leading sample eigenvector must recover nearly all of the population signal.” An outlier indicates spectral separation, not perfect directional recovery.
Correction: Evaluate recovery through $q_1=(u^\top\hat v_1)^2$. Above the threshold this overlap is positive but generally less than one; at and below the threshold it tends to zero. Population variance captured is $1+\beta q_1$, while the reconstruction loss is $p-1+\beta(1-q_1)$.
Another distinction: The sum of sample eigenvalues is empirical variance explained. It is not, in general, the population variance captured by the fitted subspace. For population reconstruction risk, use $\operatorname{tr}(\hat P_k\Sigma)$, or equivalently the exact overlap identity.

These distinctions matter whenever PCA is used for denoising, compression, or latent-signal recovery. A component can be useful for reducing reconstruction error without accurately estimating the population spike direction, and a visibly separated eigenvalue does not remove the irreducible excess risk caused by imperfect alignment.

Check Your Understanding

Use the projection-risk identity and distinguish the three variance and recovery quantities.

For a rank-$k$ projection $P$, express its population reconstruction loss using $u^\top Pu$.
Why is the oracle rank-$k$ loss $p-k$ when $k\geq1$?
If $\beta=3$, $\gamma=1$, and the leading sample-PC overlap tends to $1/2$, what are its limiting population variance captured and excess reconstruction loss?
Explain why a leading sample eigenvalue is not itself the population variance captured by its eigenvector.
What does a limiting overlap of zero imply about the spike contribution to rank-one reconstruction loss?