Worked Risk Comparisons Across Covariance Models

Why Covariance Structure Changes Interpolation Risk

An interpolating estimator can fit every training response exactly and still predict very differently across populations. The key is that prediction errors are weighted by the covariance of a new feature vector: an error in a high-variance direction matters more than an equally large coefficient error in a low-variance direction. The design also matters. Directions not represented in the training design cannot be recovered by a minimum-norm interpolator, while training noise is fitted in directions that are represented.

We will make these effects concrete with exact calculations. The examples use a deliberately transparent, coordinate-aligned design. They isolate the roles of the covariance spectrum and signal placement; they are not claims that a random design has the same risk.

Setup and an Exact Risk Decomposition

Let $p>n$, and let $X$ be a fixed $n\times p$ design matrix with full row rank. The training responses follow the linear model $y=X\beta^*+\varepsilon$, where $\beta^*\in\mathbb{R}^p$ is fixed and the training noise has mean zero and covariance matrix $\sigma^2 I_n$. The minimum-Euclidean-norm interpolator is $\hat\beta=X^\top(XX^\top)^{-1}y$. At prediction time, a fresh feature vector $x$ has mean zero and covariance $\Sigma=E[xx^\top]$, and its response is $y_{\mathrm{new}}=x^\top\beta^*+\varepsilon_{\mathrm{new}}$.

The independence condition for the fresh noise is important. We assume $E[\varepsilon_{\mathrm{new}}\mid x,\mathcal D]=0$ and $E[\varepsilon_{\mathrm{new}}^2\mid x,\mathcal D]=\tau^2$, where $\mathcal D$ denotes the training data. Independence of the fresh noise from the pair $(x,\mathcal D)$, with mean zero and variance $\tau^2$, is one sufficient condition. Merely assuming that it is independent of the training noise is not enough to remove its cross term with a prediction error that depends on $x$.

Definition: The conditional prediction risk, averaged over training noise and a fresh observation, is $R=E[(y_{\mathrm{new}}-x^\top\hat\beta)^2]$. Write $P_X=X^\top(XX^\top)^{-1}X$, the orthogonal projection onto the row space of $X$. The component $(I-P_X)\beta^*$ is the part of the signal outside that row space.

Theorem: Under the setup and assumptions above, $$ R=\tau^2+\big((I-P_X)\beta^*\big)^\top\Sigma\big((I-P_X)\beta^*\big) +\sigma^2\operatorname{tr}\!\left(\Sigma X^\top(XX^\top)^{-2}X\right). $$

The three terms are, respectively, fresh-label noise, prediction error from signal outside the design row space, and the prediction cost of fitting training noise. This is an expectation over training noise; for a particular realized training set, the fitted-noise contribution is random.

Write the estimator error.
Substitute $y=X\beta^*+\varepsilon$ into the minimum-norm formula and separate signal from training noise.

Condition on the training data.
Use the conditional mean-zero assumption for fresh noise to remove its cross term with the feature-dependent prediction error.

Average over training noise.
Use its zero mean and covariance $\sigma^2I_n$ to evaluate the quadratic noise contribution.

Proof. Define $A=X^\top(XX^\top)^{-1}$. Since $AX=P_X$, substitution gives $$ \hat\beta-\beta^*=-(I-P_X)\beta^*+A\varepsilon. $$ For a fixed training data set and feature vector, write $\delta=\hat\beta-\beta^*$. The prediction error is $\varepsilon_{\mathrm{new}}-x^\top\delta$. Conditional on $x,\mathcal D$, its squared expectation is $$ E[(\varepsilon_{\mathrm{new}}-x^\top\delta)^2\mid x,\mathcal D] =\tau^2+(x^\top\delta)^2, $$ because the conditional mean of $\varepsilon_{\mathrm{new}}$ is zero and its conditional second moment is $\tau^2$. Averaging over $x$ gives $\tau^2+\delta^\top\Sigma\delta$.

Now average the quadratic term over training noise. Put $b=(I-P_X)\beta^*$, so $\delta=-b+A\varepsilon$. The cross term has expectation zero because $E[\varepsilon]=0$: $$ E[\delta^\top\Sigma\delta] =b^\top\Sigma b+E[\varepsilon^\top A^\top\Sigma A\varepsilon]. $$ For any fixed matrix $M$, $E[\varepsilon^\top M\varepsilon]=\operatorname{tr}(M E[\varepsilon\varepsilon^\top])$. Taking $M=A^\top\Sigma A$ and using $E[\varepsilon\varepsilon^\top]=\sigma^2I_n$ yields $$ E[\varepsilon^\top A^\top\Sigma A\varepsilon] =\sigma^2\operatorname{tr}(A^\top\Sigma A) =\sigma^2\operatorname{tr}\!\left(\Sigma X^\top(XX^\top)^{-2}X\right). $$ Combining the terms proves the formula. $\square$

Coordinate-Aligned Designs: A Useful Exact Case

For the comparisons below, take $X=[I_n\ \ 0]$: the training design observes the first $n$ coordinates and no others. Let $\Sigma$ be diagonal with entries $\lambda_1,\ldots,\lambda_p$, and let the training-noise coordinates have variance $\sigma^2$. Then $P_X$ keeps the first $n$ coordinates, and $X^\top(XX^\top)^{-2}X$ has ones in the first $n$ diagonal positions and zeros elsewhere. The theorem therefore simplifies to a directly computable expression.

Corollary: For this coordinate-aligned design, $$ R=\tau^2+\sigma^2\sum_{j=1}^{n}\lambda_j+\sum_{j=n+1}^{p}\lambda_j(\beta_j^*)^2. $$

The signal term is a weighted squared norm of the unobserved coefficients. If the eigenvalues are ordered from largest to smallest, then $$ \lambda_p\sum_{j>n}(\beta_j^*)^2 \leq \sum_{j>n}\lambda_j(\beta_j^*)^2 \leq \lambda_{n+1}\sum_{j>n}(\beta_j^*)^2. $$ These bounds follow term by term from $\lambda_p\leq\lambda_j\leq\lambda_{n+1}$ for $j>n$. Thus, for the same unobserved signal norm, a smaller tail eigenvalue scale can reduce its prediction cost. The signal's location within the tail still matters.

Worked Comparisons

Worked Example: Isotropic Covariance

Take $p=6$, $n=3$, and $X=[I_3\ \ 0]$. Let $\Sigma=I_6$, $\sigma^2=0.2$, and $\tau^2=0.1$. Choose $\beta^*=(0,0,0,1,0.5,0)^\top$. The first three coefficients are observed by the design, while the last three are not.

The fitted-training-noise contribution is $$ 0.2(\lambda_1+\lambda_2+\lambda_3)=0.2(1+1+1)=0.6. $$ The signal contribution is $$ \lambda_4(1)^2+\lambda_5(0.5)^2+\lambda_6(0)^2 =1+0.25+0=1.25. $$ Adding the fresh-noise contribution gives $$ R=0.1+0.6+1.25=1.95. $$ Because the covariance is isotropic, every unobserved direction carries the same prediction cost per unit squared coefficient.

Worked Example: A Spiked Spectrum and Signal Placement

Keep $p=6$, $n=3$, and the same coordinate-aligned design, but take $\Sigma=\operatorname{diag}(4,2,1,0.5,0.25,0.25)$. Let $\sigma^2=0.2$ and $\tau^2=0.1$. First put the unobserved signal in coordinate 4: $\beta^*=(0,0,0,2,0,0)^\top$.

The training-noise term is $$ 0.2(4+2+1)=1.4, $$ and the signal term is $0.5(2)^2=2$. Hence $R=0.1+1.4+2=3.5$.

Now move the same squared signal norm, $4$, to coordinate 6: $\beta^*=(0,0,0,0,0,2)^\top$. The noise term is unchanged, but the signal term becomes $0.25(2)^2=1$, so $R=0.1+1.4+1=2.5$. The Euclidean signal norm is identical in these two cases; the lower-variance tail direction gives lower risk. For comparison, if the same norm $4$ were placed in an unobserved direction under isotropic covariance, its signal contribution would be $4$, not $1$ or $2$.

Worked Example: Rapidly Decaying Eigenvalues

Again take $p=6$, $n=3$, and $X=[I_3\ \ 0]$. Let $\Sigma=\operatorname{diag}(1,1/2,1/4,1/8,1/16,1/32)$, with $\sigma^2=0.2$ and $\tau^2=0.1$. Put a squared signal norm of $4$ in coordinate 4, so $\beta_4^*=2$ and the other coefficients are zero.

The fitted-noise contribution is $$ 0.2(1+1/2+1/4)=0.2(1.75)=0.35. $$ The signal contribution is $(1/8)(2)^2=0.5$. Therefore $$ R=0.1+0.35+0.5=0.95. $$ If the same signal norm were instead placed in coordinate 6, its contribution would be $(1/32)(2)^2=0.125$, giving $R=0.1+0.35+0.125=0.575$. The fast decay makes both the fitted-noise term in the observed coordinates and the cost of tail signal relatively small in this particular design.

Effective Dimension: A Descriptor, Not a Guarantee

One common summary of a covariance spectrum is its effective rank, $r(\Sigma)=\operatorname{tr}(\Sigma)/\lambda_1$, where $\lambda_1$ is the largest eigenvalue. For a sorted spectrum, a tail effective dimension can also be defined as $$ r_{\mathrm{tail}}(n)=\frac{\sum_{j>n}\lambda_j}{\lambda_{n+1}}, $$ when $\lambda_{n+1}>0$. These quantities describe how spread out the covariance mass is. They do not, by themselves, determine interpolation risk: the exact formula also depends on the design and on where the signal lies.

For the isotropic six-dimensional example, $r(\Sigma)=6$ and $r_{\mathrm{tail}}(3)=3$. For the spiked example, $\operatorname{tr}(\Sigma)=8$, so $r(\Sigma)=8/4=2$, and $r_{\mathrm{tail}}(3)=(0.5+0.25+0.25)/0.5=2$. For the rapidly decaying example, $\operatorname{tr}(\Sigma)=1.96875$, so $r(\Sigma)=1.96875$, and $r_{\mathrm{tail}}(3)=(0.125+0.0625+0.03125)/0.125=1.75$. These calculations summarize different spectra; they are not substitutes for the risk calculation.

In the coordinate-aligned setting, the exact formula shows why those summaries can be informative: the noise term sums eigenvalues among observed coordinates, while the signal term weights unobserved coefficients by tail eigenvalues. But a different row space, or a signal concentrated in a different eigendirection, can change the result. Benign-overfitting conclusions require conditions on the design, noise, and signal as well as on covariance decay.

Common Pitfall: Dropping a Cross Term Without the Right Assumption

A tempting but invalid argument is: “The fresh noise has mean zero and is independent of the training noise, so its cross term with prediction error vanishes.” The prediction error depends on the fresh feature $x$, and fresh noise can be associated with $x$. Independence from training noise alone does not rule that out. For example, let $U,V$ be independent Rademacher variables, take a one-dimensional fresh feature $x=U$, training noise $V$, and fresh noise $\varepsilon_{\mathrm{new}}=UV$. The fresh noise has mean zero and is independent of $V$, but $E[\varepsilon_{\mathrm{new}}x]=E[V]=0$ here; more generally, dependence on $x$ can make the cross term nonzero. The valid step is to assume $E[\varepsilon_{\mathrm{new}}\mid x,\mathcal D]=0$, as in the theorem, or impose independence of fresh noise from the pair $(x,\mathcal D)$. If that condition fails, the cross term must be calculated rather than discarded.

A second frequent overreach is to infer that a rapidly decaying spectrum always guarantees low risk. The examples establish an exact comparison for a specified coordinate-aligned design. They do not show that an arbitrary interpolator or random design has low risk just because an effective-dimension summary is small.

Key takeaway: In these exact comparisons, prediction risk separates into fresh noise, noise fitted in the design row space, and signal left outside it. Covariance eigenvalues weight both kinds of coefficient error, so spectrum and signal alignment must be considered together.

Check Your Understanding

Use the risk decomposition and distinguish conclusions specific to the coordinate-aligned examples from general claims.

For $X=[I_n\ \ 0]$ and diagonal $\Sigma$, identify which eigenvalues weight the training-noise term and which weight the signal term.
In the spiked example, why does moving a fixed squared signal norm from coordinate 4 to coordinate 6 reduce risk?
What conditional-mean assumption allows the fresh-noise cross term to vanish?
For a fixed unobserved signal norm and sorted eigenvalues, explain the upper bound on its prediction cost using $\lambda_{n+1}$.
Why does a small effective rank not, on its own, establish benign interpolation risk?