Bias–Variance Decomposition For Interpolators

Why Interpolation Does Not Determine Prediction Risk

An interpolator fits every training response exactly, but this fact alone says little about how it predicts at a new feature vector. Exact fit constrains predictions at the observed design points; away from those points, the predictor may have signal bias, may amplify noise in the training labels, or both. Prediction risk makes these sources explicit.

We work conditionally on a fixed training design. This isolates the effect of the label noise and the chosen interpolating rule. The central calculation is a squared-error identity: prediction risk is the sum of squared bias for the regression signal, variance caused by the training-label noise, and the noise in the test response that no predictor can remove.

Fixed-Design Model and Linear Predictors

Definition: Let $X\in\mathbb{R}^{n\times p}$ be a fixed design matrix with rows $x_1^T,\ldots,x_n^T$, and suppose $y=X\beta+\varepsilon$, where $\beta\in\mathbb{R}^p$ is fixed and $\varepsilon\sim N(0,\sigma^2 I_n)$. At a test feature vector $x\in\mathbb{R}^p$, let $Y_*=x^T\beta+\varepsilon_*$, where $\varepsilon_*\sim N(0,\sigma^2)$ is independent of $\varepsilon$. A linear predictor in the training responses has the form $\hat f(x)=h(x)^T y$, where $h(x)\in\mathbb{R}^n$ is fixed given $X$ and $x$. It is an interpolator if its fitted values equal $y_i$ at all training design points for every response vector $y$.

The regression signal at $x$ is $x^T\beta$; the observed test response also includes $\varepsilon_*$. Gaussian noise is a useful model here, but the decomposition of squared risk uses only the stated means, variances, and independence. The Gaussian assumption additionally makes the prediction error normally distributed.

For a fixed test point, define conditional prediction risk by $R_X(x)=E[(\hat f(x)-Y_*)^2\mid X,x]$. The expectation is over the training noise and the independent test noise, with $\beta$, $X$, and $x$ held fixed.

The Fixed-Point Risk Decomposition

Theorem: Under the fixed-design model above, for any predictor $\hat f(x)=h(x)^T y$ whose weights do not depend on the training responses, $$ R_X(x) = \underbrace{\big(h(x)^T X\beta-x^T\beta\big)^2}_{\text{squared signal bias}} + \underbrace{\sigma^2\|h(x)\|_2^2}_{\text{training-noise variance}} + \underbrace{\sigma^2}_{\text{irreducible test noise}}. $$

Proof. Substitute $y=X\beta+\varepsilon$ and $Y_*=x^T\beta+\varepsilon_*$. The prediction error is $$ \hat f(x)-Y_*= \big(h(x)^T X\beta-x^T\beta\big)+h(x)^T\varepsilon-\varepsilon_*. $$ The first term is fixed given $X,x,\beta$, while the random terms have mean zero. Independence of $\varepsilon$ and $\varepsilon_*$ gives $$ E[h(x)^T\varepsilon-\varepsilon_*]=0,\qquad \operatorname{Var}(h(x)^T\varepsilon-\varepsilon_*) =\sigma^2\|h(x)\|_2^2+\sigma^2. $$ For a random variable $Z$, $E[Z^2]=(E[Z])^2+\operatorname{Var}(Z)$. Applying this identity to the prediction error yields the displayed decomposition. $\square$

The first term concerns the mean prediction, not the observed noisy training responses. The second term measures how strongly the prediction weights transmit training-label noise: weights with a large squared norm produce greater variance. The final term remains even if the signal is predicted perfectly and training noise has no effect.

In fact, conditional on $X,x$, the error in the theorem has distribution $N(h(x)^TX\beta-x^T\beta,\ \sigma^2(\|h(x)\|_2^2+1))$. Its mean and variance agree with the three-part risk calculation.

Worked Example: Risk at a Fixed Test Point

Take $n=2$, $p=3$, and $X=\begin{pmatrix}1&0&0\\0&2&0\end{pmatrix}$, $\beta=(1,-1,2)^T$, and $\sigma^2=2$. Use the minimum-norm interpolator and evaluate risk at $x=(1,1,1)^T$.

The minimum-norm coefficient estimate is $\hat\beta=(y_1,y_2/2,0)^T$, so its prediction at $x$ is $y_1+y_2/2$. Since $E[y_1]=1$ and $E[y_2]=-2$, its expected prediction is $1-1=0$. The true test signal is $x^T\beta=1-1+2=2$, giving squared signal bias $(0-2)^2=4$.

The prediction noise is $\varepsilon_1+\varepsilon_2/2$. The two training errors are independent with variance $2$, so its variance is $2+(1/4)2=2.5$. The test response contributes irreducible noise variance $2$. Thus the total risk is $4+2.5+2=8.5$. Equivalently, the prediction error is $N(-2,4.5)$, whose second moment is $(-2)^2+4.5=8.5$.

Risk Averaged Over Test Features

Often the test feature vector is random. Let $x_*$ be independent of the training noise and test noise, with distribution $Q$ and finite second-moment matrix $M=E[x_*x_*^T]$. The matrix $M$ is a second moment, not necessarily a covariance matrix: this distinction matters if $E[x_*]\ne 0$. Averaging the fixed-point risk over $Q$ gives population prediction risk conditional on the training design.

For a coefficient estimate $\hat\beta=A y$, where $A\in\mathbb{R}^{p\times n}$ is fixed given $X$, the prediction is $x_*^T A y$. Its conditional mean coefficient is $AX\beta$, its coefficient bias is $(AX-I_p)\beta$, and its coefficient covariance is $\sigma^2AA^T$. These facts give the following result.

Theorem: If $E[\|x_*\|_2^2]<\infty$, then the risk averaged over test features satisfies $$ E[(x_*^T\hat\beta-Y_*)^2\mid X] = \beta^T(AX-I_p)^T M(AX-I_p)\beta +\sigma^2\operatorname{tr}(M AA^T) +\sigma^2. $$ Here the expectation includes $x_*$, training noise, and independent test noise.

Proof. For each fixed $x_*$, apply the fixed-point decomposition with $h(x_*)=A^Tx_*$. The signal bias is $x_*^T(AX-I_p)\beta$, and the training-noise variance is $\sigma^2x_*^TAA^Tx_*$. For any fixed vector $v$, the definition of $M$ gives $E[(x_*^Tv)^2]=v^TMv$. Applying this first with $v=(AX-I_p)\beta$, and then using $E[x_*^TAA^Tx_*]=\operatorname{tr}(AA^T M)=\operatorname{tr}(M AA^T)$, yields the two averaged terms. The irreducible term is constant in $x_*$ and remains $\sigma^2$. Finite second moments ensure these expectations exist. $\square$

For interpolation at every possible response vector, $XA=I_n$. When $X$ has full row rank, the minimum-Euclidean-norm interpolating coefficient estimate is $\hat\beta=X^T(XX^T)^{-1}y$. Substituting $A=X^T(XX^T)^{-1}$ into the theorem gives $$ E[(x_*^T\hat\beta-Y_*)^2\mid X] = \beta^T(I_p-P)^T M(I_p-P)\beta +\sigma^2\operatorname{tr}\!\left(MX^T(XX^T)^{-2}X\right) +\sigma^2, \qquad P=X^T(XX^T)^{-1}X. $$ The matrix $P$ is the orthogonal projection onto the row space of $X$. Consequently, $I_p-P$ projects onto the null space of $X$: components of $\beta$ in that null space do not affect the training signal $X\beta$, but the minimum-norm estimate sets them to zero. This is the source of its signal bias when those components matter for test prediction.

Worked Example: Minimum-Norm Risk for a Random Test Feature

Let $X=(1\ \ 0)$, $\beta=(2,3)^T$, and $\sigma^2=4$. Suppose the test-feature second moment is $M=\begin{pmatrix}1&0\\0&2\end{pmatrix}$. The minimum-norm estimator is $\hat\beta=(y_1,0)^T=(2+\varepsilon_1,0)^T$, so $A=(1,0)^T$.

Here $AX-I_2=\begin{pmatrix}0&0\\0&-1\end{pmatrix}$. Thus $(AX-I_2)\beta=(0,-3)^T$, and the signal-bias term is $(0,-3)M(0,-3)^T=18$. Also, $AA^T=\begin{pmatrix}1&0\\0&0\end{pmatrix}$, so $\sigma^2\operatorname{tr}(M AA^T)=4\operatorname{tr}\begin{pmatrix}1&0\\0&0\end{pmatrix}=4$. Adding irreducible noise gives total risk $18+4+4=26$.

The interpolator fits the sole training response exactly: its first coefficient is $y_1$. Yet the second signal coefficient, $3$, is invisible to this design and contributes $18$ to averaged squared prediction bias under the stated test-feature distribution.

Different Interpolators Can Trade Bias Against Variance

Interpolation does not by itself specify the estimator. Even with the same training design, different rules can fit the data exactly and have different prediction risks. The next calculation shows that a rule can have some signal bias while reducing variance, so comparing only whether a rule interpolates is not enough.

Worked Example: A Family of Linear Interpolators

Take $X=(1\ \ 0)$, $\beta=(1,1)^T$, and $\sigma^2=1$. The single training response is $y=1+\varepsilon$. For a fixed real number $a$, define $\hat\beta=(y,ay)^T$. At the training point $x_1=(1,0)^T$, the fitted value is $x_1^T\hat\beta=y$, so this is an interpolator.

Evaluate its risk at $x_*=(0,1)^T$. The true signal is $1$, while the predicted value is $ay$, with mean $a$ and variance $a^2$. The risk is therefore $(a-1)^2+a^2+1$, where the final $1$ is test noise variance. For $a=0$, the terms are $1,0,1$, totaling $2$. For $a=1$, they are $0,1,1$, also totaling $2$. For $a=1/2$, they are $1/4,1/4,1$, totaling $1.5$. Each rule interpolates, but the three prediction risks differ.

This example is not a claim that an arbitrary choice of $a$ is generally preferable. It demonstrates the point of the decomposition: prediction quality depends on the signal, the noise, and the test location, not merely on exact training fit.

Common Pitfall: Treating Training Fit as Zero Prediction Error

A tempting but incorrect argument is: “The interpolator has zero residuals, so its prediction risk is just the irreducible noise.” Training residuals are indeed zero, but prediction risk concerns a new response. At a test point, the expected prediction can differ from the true signal, and training noise can be carried into the prediction. The theorem shows that these contributions are respectively the squared-bias and training-variance terms; neither is forced to vanish by interpolation.

A related error is to include test noise in the variance term and then add it again as irreducible noise. Keep the sources separate: $\sigma^2\|h(x)\|_2^2$ comes from the training labels, while the final $\sigma^2$ comes from the independent test response. If the target is the noiseless signal $x^T\beta$ rather than the observed response $Y_*$, omit only the final term; the signal-bias and training-variance terms remain.

Key takeaway: Under fixed-design Gaussian regression, squared prediction risk separates exactly into signal bias, variance transmitted from training-label noise, and irreducible test noise. Interpolation controls training fit, not the values of those three prediction-risk components.

Check Your Understanding

Use the fixed-design model and distinguish prediction of the noiseless signal from prediction of the noisy test response.

For a fixed test point, what is the training-noise variance when the prediction weights are $h(x)$?
Why does an interpolator’s zero training residual not imply zero signal bias at a new point?
In the averaged-risk formula, why is $M=E[x_*x_*^T]$ used rather than necessarily the covariance matrix of $x_*$?
For $X=(1\ \ 0)$, what part of $\beta=(\beta_1,\beta_2)^T$ is not identified by the training signal?
If prediction risk is defined against the noiseless target $x_*^T\beta$, which term in the three-part decomposition is removed?