Anisotropic Spectra And Benign Overfitting

Why Anisotropy Matters After Interpolation

An interpolating estimator has zero training error, but that fact alone says little about its prediction error. In linear regression, prediction risk depends on which directions in predictor space the training design captures, how much test-predictor variation lies in those directions, and where the regression signal lies. When predictor directions have different variances, these features cannot be summarized by the number of parameters and observations alone.

This tutorial studies the minimum-Euclidean-norm interpolator in the overparameterized regime. The main calculation is conditional on the observed design matrix, so it applies to a fixed design and also supplies a starting point for averaging over a random design. It separates two effects: signal left in directions the data do not identify, and training noise amplified when the estimator fits it exactly.

Definition: Let $X$ be the $n\times p$ design matrix, with row $i$ equal to $x_i^T$, where $p>n$ and $\operatorname{rank}(X)=n$. Suppose $y=X\beta^*+\varepsilon$, where $\beta^*\in\mathbb{R}^p$ is the signal and, conditional on $X$, the noise satisfies $E[\varepsilon\mid X]=0$ and $\operatorname{Cov}(\varepsilon\mid X)=\sigma^2 I_n$. For an independent test predictor $x_{\mathrm{new}}$ with mean zero and covariance $\Sigma=E[x_{\mathrm{new}}x_{\mathrm{new}}^T]$, define the excess prediction risk of an estimator $\hat\beta$ by $E[(x_{\mathrm{new}}^T(\hat\beta-\beta^*))^2\mid X,\varepsilon]$. The minimum-Euclidean-norm interpolator is $\hat\beta=X^T(XX^T)^{-1}y$; it interpolates because $X\hat\beta=y$.

The test-predictor distribution is assumed independent of the training noise and training observations. The covariance $\Sigma$ is positive semidefinite; when it has eigenvalues $\lambda_j$, those eigenvalues measure prediction importance in their respective eigen-directions. The risk here is excess squared prediction error for the conditional mean. If test responses also include independent noise with variance $\sigma^2$, their mean squared prediction error has an additional irreducible $\sigma^2$ term.

Conditional Bias and Variance

Write $P=X^T(XX^T)^{-1}X$. Since $X$ has full row rank, $P$ is the orthogonal projection onto the row space of $X$, and $I_p-P$ projects onto its null space. Substituting the linear model into the estimator gives $\hat\beta=P\beta^*+X^T(XX^T)^{-1}\varepsilon$. Thus the signal component is projected onto the observed row space, while the noise component is fitted there as well.

Theorem: Under the preceding assumptions, the expected excess prediction risk conditional on $X$, averaging over training noise, is $$ E_{\varepsilon}\!\left[E_{x_{\mathrm{new}}}\!\left[(x_{\mathrm{new}}^T(\hat\beta-\beta^*))^2\mid X,\varepsilon\right]\middle|X\right] = \underbrace{\beta^{*T}(I_p-P)\Sigma(I_p-P)\beta^*}_{\text{conditional squared bias}} + \underbrace{\sigma^2\operatorname{tr}\!\left(\Sigma X^T(XX^T)^{-2}X\right)}_{\text{conditional variance}}. $$

Proof. For any fixed vector $a$, the mean-zero test-predictor assumption gives $E[(x_{\mathrm{new}}^Ta)^2]=a^T\Sigma a$. Conditional on $X$, write the coefficient error as $\hat\beta-\beta^*=-(I_p-P)\beta^*+A\varepsilon$, where $A=X^T(XX^T)^{-1}$. Its conditional mean is $-(I_p-P)\beta^*$, and its conditional covariance is $\sigma^2AA^T=\sigma^2X^T(XX^T)^{-2}X$. For any random vector $Z$ with finite second moments, $E[Z^T\Sigma Z]=(EZ)^T\Sigma(EZ)+\operatorname{tr}(\Sigma\operatorname{Cov}(Z))$; this follows by expanding $Z=(EZ)+(Z-EZ)$ and using that the centered cross terms have expectation zero. Applying this identity to the coefficient error yields the displayed bias and variance. $\square$

This is an exact finite-sample identity for the stated fixed design. It does not assume that $X$ is isotropic, nor that its row space is aligned with the eigenvectors of $\Sigma$. If the design is random, its unconditional risk is obtained by averaging the two conditional terms over the distribution of $X$, when those expectations exist.

Worked Example: Signal Hidden in a Low-Variance Direction

Let $p=3$, $n=2$, and let the training rows be $e_1^T$ and $e_2^T$, where $e_j$ is coordinate vector $j$. Take no training noise, $\beta^*=(1,2,3)^T$, and $\Sigma=\operatorname{diag}(4,1,0.01)$. The design has $P=\operatorname{diag}(1,1,0)$, so the minimum-norm interpolator is $\hat\beta=P\beta^*=(1,2,0)^T$. It fits both training responses exactly.

The coefficient error is $(- (I_p-P)\beta^*)=(0,0,-3)^T$. Consequently the excess prediction risk is $(0,0,-3)\Sigma(0,0,-3)^T=9(0.01)=0.09$. If instead the signal were $(1,2,0)^T$, with everything else unchanged, the estimator would recover it exactly and the excess risk would be zero. The difference is signal alignment: the unobserved coefficient has squared prediction cost equal to its squared size multiplied by the variance of its direction.

Reading the Risk in Eigen-Directions

Let $u_1,\ldots,u_p$ be an orthonormal eigenbasis of $\Sigma$, with $\Sigma u_j=\lambda_j u_j$ and $\lambda_j\geq 0$. Expanding the quadratic form gives $$ \beta^{*T}(I_p-P)\Sigma(I_p-P)\beta^* =\sum_{j=1}^p\lambda_j\bigl(u_j^T(I_p-P)\beta^*\bigr)^2. $$ Thus an omitted component matters in proportion to both its size after projection onto the null space and the covariance eigenvalue weighting that component in prediction. When the row-space projection does not preserve the covariance eigen-directions, these projected components can mix signal coordinates; it is not generally valid to treat each original coordinate as simply observed or omitted.

The variance term has a parallel spectral expression: $$ \sigma^2\operatorname{tr}\!\left(\Sigma X^T(XX^T)^{-2}X\right) =\sigma^2\sum_{j=1}^p\lambda_j\left\|(XX^T)^{-1}Xu_j\right\|^2. $$ This follows by writing $\Sigma=\sum_j\lambda_j u_ju_j^T$ inside the trace and using $u_j^TX^T=(Xu_j)^T$. It shows that a direction can contribute substantial variance when it has high test-predictor variance and the observed design requires a large coefficient response to fit noise in that direction. Eigenvalues alone do not determine this response: the realized design matrix also matters.

Derived result: Suppose the training rows are $s_j e_j^T$ for $j$ in an observed index set $J$, with $s_j\ne0$, and $\Sigma$ is diagonal with entries $\lambda_j$. Then the minimum-norm interpolator sets unobserved coordinates to zero, and its conditional expected excess risk is $$ \sum_{j\notin J}\lambda_j\beta_j^{*2} +\sigma^2\sum_{j\in J}\frac{\lambda_j}{s_j^2}. $$

Derivation. Each training equation is $y_j=s_j\beta_j^*+\varepsilon_j$. The minimum-norm solution has $\hat\beta_j=y_j/s_j=\beta_j^*+\varepsilon_j/s_j$ for $j\in J$, and $\hat\beta_j=0$ otherwise. For an unobserved coordinate the error is $-\beta_j^*$; its prediction contribution is $\lambda_j\beta_j^{*2}$. For an observed coordinate, the error is $\varepsilon_j/s_j$, whose mean is zero and variance is $\sigma^2/s_j^2$, so its expected contribution is $\lambda_j\sigma^2/s_j^2$. Diagonality makes the risk the sum of these contributions. This also proves the formula in the info box. $\square$

Worked Example: Noise Amplification in an Observed Direction

Use three coordinates, observe the first two with training rows $2e_1^T$ and $0.5e_2^T$, and take $\Sigma=\operatorname{diag}(3,2,0.1)$, $\beta^*=(1,-1,2)^T$, and $\sigma^2=0.04$. The third coordinate is omitted, so its bias contribution is $0.1(2^2)=0.4$.

For coordinate 1, the fitted coefficient is $1+\varepsilon_1/2$. Its variance contribution is $3(0.04/2^2)=3(0.01)=0.03$. For coordinate 2, the fitted coefficient is $-1+\varepsilon_2/0.5=-1+2\varepsilon_2$. Its variance contribution is $2(0.04/0.5^2)=2(0.16)=0.32$. The total conditional expected excess risk is therefore $0.4+0.03+0.32=0.75$. The estimator still interpolates exactly: the larger noise cost in coordinate 2 comes from the smaller design multiplier $s_2=0.5$, which requires greater coefficient adjustment to fit a given noise value.

How Spectrum and Alignment Can Favor Interpolation

The exact decomposition identifies a mechanism behind benign overfitting: an interpolator can fit noisy training responses yet have modest prediction risk if the signal it fails to recover is prediction-irrelevant enough and the noise-fitting component has limited prediction cost. In an eigen-aligned example, leaving a signal coefficient in a direction with small $\lambda_j$ incurs small bias, since that direction varies little in the test population. But fitting noise in a direction with a small design multiplier can be costly: its variance contribution contains $1/s_j^2$. The same low-variance direction can therefore be cheap to omit and expensive to fit, depending on the design and the part of the error being considered.

Worked Example: Choosing Different Directions Changes the Tradeoff

Keep $\Sigma=\operatorname{diag}(3,2,0.1)$, $\beta^*=(1,-1,2)^T$, and $\sigma^2=0.04$, but now observe coordinates 2 and 3 with rows $2e_2^T$ and $0.5e_3^T$. Coordinate 1 is omitted, giving bias $3(1^2)=3$. The variance contributions are $2(0.04/2^2)=0.02$ for coordinate 2 and $0.1(0.04/0.5^2)=0.016$ for coordinate 3. The total is $3+0.02+0.016=3.036$.

The earlier design had total risk $0.75$, whereas this design has risk $3.036$, despite both interpolating two observations with two fitted directions. This comparison changes the observed design directions as well as their multipliers; it is an illustration of the risk decomposition, not a claim that either design is typical of a particular random sampling model. The high-variance first direction carries signal that the second design leaves unrecovered.

Common Mistake: Treating Small Eigenvalues as a Guarantee

Wrong reasoning: “The covariance has small eigenvalues, so minimum-norm interpolation must generalize well.” Small eigenvalues can reduce the prediction cost of signal left in those directions, but the risk formula has a separate variance term. If the design weakly captures a direction, fitting its noise may require a large coefficient, and the resulting prediction variance can be large. Moreover, if the signal is concentrated in a high-eigenvalue direction that the design misses, the bias can be substantial.

Correction: Inspect both terms, with the design and signal specified. Ask how the null-space projection acts on $\beta^*$, how $\Sigma$ weights that error, and how the inverse of $XX^T$ amplifies noise in directions relevant to test prediction. A population spectrum by itself is not a complete risk calculation. In random-design problems, properties of the joint relationship between the spectrum and the sampled row space also matter.

Key takeaway: For minimum-norm interpolation, anisotropic prediction risk is a design-dependent balance between unobserved signal and fitted noise. Covariance eigenvalues weight prediction errors, signal alignment controls bias, and design strength controls variance. Interpolation is compatible with small prediction risk, but it does not guarantee it.

Check Your Understanding

Use the risk decomposition to reason through these questions.

In the exact conditional formula, what projection determines which part of the signal contributes to bias?
For diagonal covariance and an aligned design, why does omitting a coefficient of fixed size cost less in a direction with a smaller eigenvalue?
In the derived coordinate-wise formula, what happens to the noise contribution as an observed design multiplier $s_j$ becomes smaller?
Can covariance eigenvalues alone determine the variance of the minimum-norm interpolator? Explain which other quantity appears in the variance term.
Why does exact interpolation not imply zero excess prediction risk?