Minimum-Norm Interpolation Under Isotropic Design

Setup: Isotropic Features and Prediction Risk

Minimum-norm interpolation gives an exact fit to the training responses when there are fewer observations than features. Exact fit, however, does not by itself determine how well the fitted rule predicts new observations. Under an isotropic Gaussian design, symmetry makes the risk calculable: it separates into error from signal directions that the data cannot identify and error contributed by label noise.

Let $x_1,\ldots,x_n$ be independent $N(0,I_p)$ feature vectors, and let $X$ be the $n$-by-$p$ matrix whose $i$th row is $x_i^\top$. The response model is $y=X\beta^\star+\varepsilon$, where $\beta^\star\in\mathbb R^p$ is fixed and $\varepsilon\in\mathbb R^n$ has independent, mean-zero entries of variance $\sigma^2$, independent of $X$. The noise need not be Gaussian for the risk calculations below.

For an independent test feature $x_0\sim N(0,I_p)$, define the excess prediction risk conditional on the training data as the squared prediction error relative to the regression function $x_0^\top\beta^\star$. Its expectation over the training data is the expected excess prediction risk. This tutorial reports excess risk; if predicting a future response with independent noise of variance $\sigma^2$, add $\sigma^2$ to obtain its mean squared prediction error.

Definition: The minimum-norm least-squares estimator is a vector minimizing $\|y-Xb\|^2$ and, among all such minimizers, having the smallest Euclidean norm. When $p>n$, $X$ has full row rank with probability one and the estimator interpolates: $$ \hat\beta=X^\top(XX^\top)^{-1}y. $$ When $n>p$, $X$ has full column rank with probability one and the least-squares estimator is unique: $$ \hat\beta=(X^\top X)^{-1}X^\top y. $$

For any fixed coefficient error $v$, isotropy means $E[(x_0^\top v)^2]=\|v\|^2$. Consequently, conditional on the training data, the excess prediction risk is $\|\hat\beta-\beta^\star\|^2$. This identity turns the prediction calculation into a coefficient-error calculation.

Overparameterized Interpolation: Signal and Noise

Suppose $p>n$. With probability one, the rows of $X$ are linearly independent. Define $P=X^\top(XX^\top)^{-1}X$, the orthogonal projection onto the row space of $X$. Substituting the response model into the interpolator gives $\hat\beta=P\beta^\star+X^\top(XX^\top)^{-1}\varepsilon$. Thus $\hat\beta-\beta^\star=-(I_p-P)\beta^\star+X^\top(XX^\top)^{-1}\varepsilon$. The first term lies in the null space of $X$; the second lies in its row space. They are orthogonal.

Theorem: Under the stated Gaussian isotropic design and independent mean-zero noise with variance $\sigma^2$, if $p\ge n+2$, the expected excess prediction risk of the minimum-norm interpolator is $$ E\big[\|\hat\beta-\beta^\star\|^2\big] = \left(1-\frac{n}{p}\right)\|\beta^\star\|^2 +\sigma^2\frac{n}{p-n-1}. $$ The first term is signal error from the unobserved null space; the second is noise error. If $p=n$ or $p=n+1$, the expected risk is infinite when $\sigma^2>0$.

Proof. Conditional on $X$, the two orthogonal components yield $$ \|\hat\beta-\beta^\star\|^2 = \|(I_p-P)\beta^\star\|^2 +\|X^\top(XX^\top)^{-1}\varepsilon\|^2. $$ The conditional expectation of the second term over the noise is $\sigma^2\operatorname{tr}((XX^\top)^{-1})$: use $E[\varepsilon\varepsilon^\top]=\sigma^2 I_n$ and the identity $\|X^\top(XX^\top)^{-1}\varepsilon\|^2 =\varepsilon^\top(XX^\top)^{-1}\varepsilon$.

To average the first term, Gaussian rotational invariance implies that $E[P]$ is unchanged by conjugation by every orthogonal $p$-by-$p$ matrix. Hence $E[P]$ is a scalar multiple of $I_p$. Since $P$ has rank $n$, $\operatorname{tr}(E[P])=E[\operatorname{tr}(P)]=n$, so $E[P]=(n/p)I_p$. Therefore $E[\|(I_p-P)\beta^\star\|^2]=(1-n/p)\|\beta^\star\|^2$.

For the noise term, write the rows of $X$ as $x_1^\top,\ldots,x_n^\top$. The diagonal entry $((XX^\top)^{-1})_{ii}$, by the Schur complement formula, is the reciprocal of the squared length of the component of $x_i$ orthogonal to the span of the other $n-1$ rows. Conditional on those rows, that squared length has a chi-squared distribution with $p-n+1$ degrees of freedom. If $Z$ is chi-squared with $k>2$ degrees of freedom, its density gives $E[1/Z]=1/(k-2)$: integrating the chi-squared density and using $\Gamma(k/2)=(k/2-1)\Gamma(k/2-1)$ gives this value. Thus each diagonal entry has expectation $1/(p-n-1)$, and $E[\operatorname{tr}((XX^\top)^{-1})]=n/(p-n-1)$.

When $p=n$ or $p=n+1$, the corresponding residual squared length has respectively one or two degrees of freedom. Its reciprocal has infinite expectation, so the trace expectation, and hence the noise risk for positive $\sigma^2$, is infinite. Combining the two components proves the stated formula and boundary behavior. $\square$

Worked Risk Calculations

Worked Example: More Features Than Observations

Suppose $n=40$, $p=100$, $\|\beta^\star\|^2=4$, and $\sigma^2=1$. Since $100\ge40+2$, the finite-risk formula applies.

The signal contribution is $(1-40/100)4=0.6(4)=2.4$. The noise contribution is $1\cdot40/(100-40-1)=40/59\approx0.678$. Therefore the expected excess prediction risk is $2.4+40/59\approx3.078$.

Even though the training responses are fit exactly, the signal contribution remains: the component of $\beta^\star$ orthogonal to the observed row space cannot be recovered by this estimator.

Worked Example: More Observations Than Features

Now take $n=100$, $p=40$, $\|\beta^\star\|^2=4$, and $\sigma^2=0.25$. The model is correctly specified, and $n\ge p+2$, so the unique least-squares estimator has finite expected risk.

Substituting $y=X\beta^\star+\varepsilon$ into the estimator gives $\hat\beta-\beta^\star=(X^\top X)^{-1}X^\top\varepsilon$. There is no null-space signal component because $X$ has full column rank. The expected excess risk is $\sigma^2 p/(n-p-1)=0.25(40)/(100-40-1)=10/59\approx0.1695$.

The value $4$ for the signal norm does not enter this risk: in this full-rank, correctly specified model, the estimator has no signal bias.

Worked Example: A Small Overparameterization Gap

Let $n=50$, $p=53$, $\|\beta^\star\|^2=9$, and $\sigma^2=0.04$. The denominator $p-n-1=2$ is positive, so the expected risk is finite.

The null-space signal contribution is $(1-50/53)9=(3/53)9=27/53\approx0.5094$. The noise contribution is $0.04(50/2)=1$. The total expected excess risk is $27/53+1\approx1.5094$.

The small dimension gap makes the noise term large relative to the noise variance: it is $25$ times $0.04$. This example is finite, unlike the case $p=n+1$, where the expected inverse-Gram contribution diverges.

Why the Dimension Regime Matters

The overparameterized formula applies to interpolation. When $n>p$, the least-squares estimator is generally not an interpolator in the presence of noise; it minimizes residual sum of squares instead. Its risk nevertheless provides a useful comparison. The estimator has no signal bias under the correctly specified model, and, when $n\ge p+2$, $$ E\big[\|\hat\beta-\beta^\star\|^2\big] = \sigma^2\frac{p}{n-p-1}. $$ To see the variance factor, condition on $X$: the coefficient error has covariance $\sigma^2(X^\top X)^{-1}$, so its expected squared norm is $\sigma^2\operatorname{tr}((X^\top X)^{-1})$. Applying the same Schur-complement argument to the $p$ independent Gaussian columns of $X$, viewed as vectors in $\mathbb R^n$, gives expected diagonal entries $1/(n-p-1)$. Summing $p$ entries yields the expression.

When $n=p+1$, that inverse moment is infinite; when $n=p$, the square design also has infinite expected inverse-Gram trace. Thus a formula with a denominator approaching zero is not merely a numerical warning: at the boundary, the expectation itself does not exist as a finite number.

Regime	Estimator behavior	Expected excess risk when finite
$p\ge n+2$	Minimum-norm interpolator	$(1-n/p)\\|\beta^\star\\|^2+\sigma^2n/(p-n-1)$
$n\ge p+2$	Unique least-squares estimator	$\sigma^2p/(n-p-1)$
$\|p-n\|\le1$	Near the square-design boundary	Infinite expected risk if $\sigma^2>0$

The isotropic assumption is central to these concise expressions. It makes prediction risk equal to Euclidean coefficient error and makes the random row space directionally symmetric. With unequal feature variances, coefficient directions no longer have equal predictive weight; that requires a different risk analysis.

Common Pitfall: Exact Fit Does Not Mean Zero Risk

A common incorrect argument is: “The interpolator satisfies $X\hat\beta=y$, so its prediction error must be zero.” This confuses training residuals with error on an independent test feature. Interpolation only forces agreement at the observed design points. When $p>n$, infinitely many coefficient vectors fit those points; the minimum-norm choice still cannot recover the part of $\beta^\star$ in the null space of $X$. Moreover, it fits the training noise, which contributes variance to predictions.

The correction is to calculate prediction error on a fresh $x_0$, and then average over the training data if the target is expected risk. In the isotropic model, those steps give the signal term $(1-n/p)\|\beta^\star\|^2$ and the noise term $\sigma^2 n/(p-n-1)$, rather than zero.

Key takeaway: In isotropic Gaussian regression, minimum-norm interpolation has two distinct sources of excess risk: unrecovered signal in the unobserved directions and amplified label noise. Sample size and dimension determine both, while signal norm and noise variance set their respective scales.

Check Your Understanding

Use the assumptions and formulas stated above to answer these questions.

For $p>n$, why are the signal and noise components of the coefficient error orthogonal?
With $n=30$, $p=50$, $\|\beta^\star\|^2=2$, and $\sigma^2=0.5$, calculate both contributions to expected excess risk.
Why does the isotropic test-feature assumption make prediction risk equal to squared coefficient error?
What happens to expected noise risk when $p=n+1$, and why is the risk infinite even though the design matrix has full row rank with probability one?
In the regime $n>p$, why does the correctly specified least-squares estimator have no signal-bias term?

Regime	Estimator behavior	Expected excess risk when finite
\(p\ge n+2\)	Minimum-norm interpolator	\((1-n/p)\\|\beta^\star\\|^2+\sigma^2n/(p-n-1)\)
\(n\ge p+2\)	Unique least-squares estimator	\(\sigma^2p/(n-p-1)\)
\(\|p-n\|\le1\)	Near the square-design boundary	Infinite expected risk if \(\sigma^2>0\)