Diagnosing A Benign-Overfitting Claim

Exact Fit Is Not a Risk Guarantee

An interpolating estimator has zero empirical training error, but that fact alone says little about how it predicts on new observations. A claim of benign overfitting is therefore a claim about a specified population risk under specified assumptions: the loss, the noise, the design distribution, the signal, and the relation between sample size and dimension all matter. A calculation under squared prediction loss cannot, without additional conditions, establish a guarantee for a different loss.

We will audit claims in the linear regression setting. The purpose is not to assert one universal benign-overfitting theorem: the sufficient conditions depend on the model and the guarantee being claimed. Instead, we derive the exact conditional risk identity and use it to identify what a valid argument must check.

Definition: Let \(X\) be an \(n\times p\) design matrix with rows \(x_i^T\), and let \(y=X\beta^*+\varepsilon\). When \(n<p\) and \(X\) has full row rank, the minimum-Euclidean-norm interpolator is \(\hat\beta=X^T(XX^T)^{-1}y\); it satisfies \(X\hat\beta=y\). For a fresh covariate \(x\), define its squared prediction excess risk, conditional on the training design and responses, as \(R(\hat\beta)=E_x[(x^T\hat\beta-x^T\beta^*)^2]\), where \(E[xx^T]=\Sigma\) and \(E_x\) averages over a fresh draw from that covariate distribution.

The risk here is for predicting the conditional mean \(x^T\beta^*\), not for predicting a future noisy response. If the future response is \(Y=x^T\beta^*+\varepsilon_{\mathrm{new}}\), its expected squared prediction error has the additional irreducible noise variance when that variance is finite and the new noise is mean zero and independent. The distinction matters when a claim uses the word “risk” without specifying its target.

The Conditional Risk Identity

Assume \(X\) is fixed, has full row rank, and \(n<p\). Suppose the training noise is independent of \(X\), has \(E[\varepsilon\mid X]=0\) and \(\operatorname{Cov}(\varepsilon\mid X)=\sigma^2 I_n\), and the fresh covariate has finite second moment \(\Sigma=E[xx^T]\). No Gaussian assumption is needed for the identity below.

Theorem: Under these conditions, the risk averaged over training noise, conditional on \(X\), is \(E_{\varepsilon}[R(\hat\beta)\mid X]=\|(I-P_X)\beta^*\|_{\Sigma}^2+\sigma^2\operatorname{tr}\{\Sigma X^T(XX^T)^{-2}X\}\), where \(P_X=X^T(XX^T)^{-1}X\), and \(\|v\|_\Sigma^2=v^T\Sigma v\). The first term is signal bias from the unobserved parameter directions; the second is noise amplification.

Proof. Define \(A=X^T(XX^T)^{-1}\). Substituting \(y=X\beta^*+\varepsilon\) gives \(\hat\beta=A X\beta^*+A\varepsilon=P_X\beta^*+A\varepsilon\), since \(AX=P_X\). Thus \(\hat\beta-\beta^*=-(I-P_X)\beta^*+A\varepsilon\). For any fixed vector \(v\), \(E_x[(x^Tv)^2]=v^T\Sigma v\), by expanding the square and using \(E[xx^T]=\Sigma\). Therefore \(R(\hat\beta)=(\hat\beta-\beta^*)^T\Sigma(\hat\beta-\beta^*)\).

Now average this quadratic form over \(\varepsilon\). Its cross term has expectation zero because \(E[\varepsilon\mid X]=0\). The remaining noise term is \(E[(A\varepsilon)^T\Sigma(A\varepsilon)\mid X]=\operatorname{tr}\{\Sigma A E[\varepsilon\varepsilon^T\mid X]A^T\}=\sigma^2\operatorname{tr}(\Sigma AA^T)\). Since \(AA^T=X^T(XX^T)^{-2}X\), the claimed expression follows. \(\square\)

This identity gives a practical checklist. The bias term depends on how the signal lies relative to the observed row space, measured in the population covariance geometry. The variance term depends on both \(\Sigma\) and the inverse of the sample Gram matrix \(XX^T\). Small sample eigenvalues can strongly amplify noise. Neither term is determined merely by \(n<p\) or by exact training fit.

Worked Risk Audits

Worked Example: A Signal Hidden in a Poorly Weighted Direction

Take \(n=1\), \(p=2\), design row \(X=(1,1)\), population covariance \(\Sigma=\begin{pmatrix}1&0\\0&9\end{pmatrix}\), signal \(\beta^*=(1,0)^T\), and mean-zero training noise with variance \(\sigma^2=0.04\). Compute the conditional noise-averaged prediction risk.

Here \(XX^T=2\), so \(P_X=X^T(XX^T)^{-1}X=\frac12\begin{pmatrix}1&1\\1&1\end{pmatrix}\). Consequently, \(P_X\beta^*=(1/2,1/2)^T\), and \((I-P_X)\beta^*= (1/2,-1/2)^T\). The squared signal-bias contribution is \((1/2,-1/2)\Sigma(1/2,-1/2)^T=1/4+9/4=2.5\).

Also \(X^T(XX^T)^{-2}X=\frac14\begin{pmatrix}1&1\\1&1\end{pmatrix}\). Multiplying by \(\Sigma\) and taking the trace gives \((1+9)/4=2.5\), so the noise contribution is \(0.04(2.5)=0.1\). The total risk averaged over training noise is therefore \(2.5+0.1=2.6\).

The example interpolates exactly, but its risk is not small. The signal component not captured by the row space contributes substantially, especially because the second coordinate receives high population weight. By contrast, if the signal were \((1,1)^T\), it would lie in the row space and the bias term would be zero. Alignment must be assessed together with the population covariance, not by signal norm alone.

Worked Example: A Small Sample Singular Value Amplifies Noise

Let \(n=1\), \(p=2\), \(X=(0.1,0)\), \(\Sigma=I_2\), and \(\beta^*=0\). Let the training noise have mean zero and variance \(0.01\). Find the expected squared prediction risk.

The fitted coefficient is \(\hat\beta=X^T(XX^T)^{-1}\varepsilon=(0.1,0)^T(0.01)^{-1}\varepsilon=(10\varepsilon,0)^T\). Since \(\Sigma=I_2\), the conditional prediction risk for a realized \(\varepsilon\) is \(\|\hat\beta\|^2=100\varepsilon^2\). Averaging over noise gives \(100(0.01)=1\).

For comparison, replacing the row by \(X=(1,0)\) gives \(\hat\beta=(\varepsilon,0)^T\) and risk \(E[\varepsilon^2]=0.01\). Both designs have full row rank and both interpolate. The first has a much smaller nonzero singular value, so inversion magnifies the noise. A sample-size condition such as \(n<p\) does not control this amplification.

Result: If \(\Sigma=I_p\) and the noise has covariance \(\sigma^2I_n\), then for any full-row-rank \(X\), the conditional noise contribution is \(\sigma^2\operatorname{tr}\{(XX^T)^{-1}\}\), which is at least \(\sigma^2/\lambda_{\min}(XX^T)\).

Proof. With \(\Sigma=I_p\), the variance term in the theorem is \(\sigma^2\operatorname{tr}\{X^T(XX^T)^{-2}X\}\). The cyclic trace identity gives \(\operatorname{tr}\{X^T(XX^T)^{-2}X\}=\operatorname{tr}\{(XX^T)^{-2}XX^T\}=\operatorname{tr}\{(XX^T)^{-1}\}\). Let the positive eigenvalues of \(XX^T\) be \(\lambda_1,\ldots,\lambda_n\). The trace is \(\sum_{i=1}^n1/\lambda_i\), which is at least \(1/\lambda_{\min}(XX^T)\). Multiplying by \(\sigma^2\) proves the result. \(\square\)

Worked Example: A Nonzero Noise Mean Changes the Answer

Take \(X=(1,1)\), \(\Sigma=I_2\), and \(\beta^*=0\). Suppose the one training noise value has mean \(1\) and variance \(0.25\). The fitted coefficient is \(\hat\beta=(\varepsilon/2,\varepsilon/2)^T\), so its squared prediction risk is \(\|\hat\beta\|^2=\varepsilon^2/2\). Since \(E[\varepsilon^2]=\operatorname{Var}(\varepsilon)+(E[\varepsilon])^2=0.25+1=1.25\), the expected risk is \(1.25/2=0.625\).

If the noise instead had mean zero but the same variance, the expected risk would be \(0.25/2=0.125\). The difference is not a harmless technicality: the zero-mean theorem’s variance term alone would miss the contribution from the nonzero mean. A guarantee must state whether noise is centered, and whether that condition holds given the covariates. Likewise, a guarantee for squared prediction risk does not automatically bound classification error or another loss.

What a Benign-Overfitting Claim Must Specify

A proposed guarantee is credible only when its assumptions control both terms in the risk identity, or otherwise prove the claimed risk bound directly. A useful audit proceeds in order:

Name the risk and loss.
State whether the target is squared error for the conditional mean, squared error for a future response, or a different loss. Do not transfer a bound between losses without a result that justifies the transfer.

Check the noise model.
Record whether noise is centered conditional on the design, independent across observations, and homoskedastic with finite variance. If its covariance is not \(\sigma^2I_n\), use that covariance in the variance calculation rather than silently substituting the homoskedastic formula.

Inspect covariance geometry.
Identify the population covariance and the directions its prediction risk weights. Check whether the assumptions control the relevant covariance spectrum and how it interacts with the sample Gram matrix.

Check signal alignment.
Determine whether the unobserved component \((I-P_X)\beta^*\) is negligible in the \(\Sigma\)-weighted norm, or whether a stated structural condition controls it. A small Euclidean norm by itself may not suffice when covariance directions have very different weights.

Verify the sample-size regime.
Confirm the required relation between \(n\) and \(p\), full row rank or its replacement, and any additional spectral or effective-dimension conditions. If the guarantee is asymptotic, identify the sequence of models and the limit in which the risk is claimed to stay bounded or vanish.

Population covariance eigenvalues and sample Gram-matrix eigenvalues are related but not interchangeable. The former describe how a fresh covariate weights prediction errors; the latter determine how the observed design is inverted to interpolate. A rigorous guarantee has to connect these objects under its stated design assumptions. Merely citing a favorable population spectrum does not establish that a particular sample matrix is well-conditioned, and merely observing a well-conditioned sample matrix does not settle the signal-bias term.

Common Mistakes

“The estimator fits every training point, so its prediction risk is zero.” Interpolation says \(X\hat\beta=y\); population risk averages over fresh \(x\). The two quantities are different.
“There are more parameters than observations, so the minimum-norm fit is benign.” The inequality \(n<p\) is not a control on the signal bias or on small singular values, as the examples show.
“The noise variance is small, so the risk must be small.” The variance contribution also includes the inverse design geometry. A small variance can still be amplified by a poorly conditioned sample.
“A result for squared prediction risk proves good performance under any loss.” The result concerns the loss it defines. A guarantee for a different loss requires its own argument and assumptions.

Key takeaway: Treat benign overfitting as a conditional, model-specific risk claim—not as a synonym for interpolation. Check the loss, noise, covariance spectrum, signal alignment, and sample-size regime, then verify that the argument controls the resulting prediction risk.

Check Your Understanding

Use the risk identity and the assumptions stated in each question.

Why can an estimator have zero training error but positive squared prediction risk?
In the conditional risk theorem, which assumption makes the cross term between signal bias and fitted noise have expectation zero?
For \(\Sigma=I_p\), why does a small \(\lambda_{\min}(XX^T)\) pose a risk for interpolation?
What part of the risk identity changes if the signal lies in the row space of \(X\)?
Why is a squared prediction-risk guarantee not by itself a guarantee for classification error?