Interpolation And Prediction Risk

Training Fit and Prediction Are Different Quantities

An estimator can match every response in its training data and still predict poorly at new feature values. Conversely, an exact fit can have zero prediction risk when the fitted rule agrees with the population relationship. To distinguish these cases, specify which observations are averaged over: the observed training sample or a fresh draw from the population.

Consider supervised data $Z_i=(X_i,Y_i)$, $i=1,\ldots,n$, drawn independently from a distribution $P$. A predictor $f$ maps a feature value $x$ to a response prediction $f(x)$. In a learning problem, $f$ may itself depend on the training data. The population prediction risk is evaluated on a new pair $(X,Y)$ independent of the training sample.

Definition: Under squared loss, the empirical risk of a predictor $f$ on a training sample is $$ R_n(f)=\frac{1}{n}\sum_{i=1}^{n}(Y_i-f(X_i))^2. $$ Its population prediction risk is $$ R(f)=E[(Y-f(X))^2], $$ where $(X,Y)\sim P$ is a fresh observation. A predictor is an interpolator for the sample if $R_n(f)=0$. For a data-dependent estimator $\hat f$, these quantities are evaluated at the realized fitted predictor; in particular, $R(\hat f)$ is conditional on the training data unless an expectation over the training sample is explicitly taken.

Because each term in $R_n(f)$ is nonnegative, $R_n(f)=0$ if and only if $f(X_i)=Y_i$ for every observed training pair. This is an exact statement about those $n$ pairs. Population risk averages over the distribution of all possible pairs, including feature values that may not occur in the sample. No equality between these two averages follows just from interpolation.

Worked Examples

Worked Example: A Continuous Feature and Arbitrarily Poor Prediction

Suppose the training data consist of $(X_1,Y_1)=(0,0)$ and $(X_2,Y_2)=(1,0)$. In the population, $X$ is uniform on $[0,1]$ and $Y=0$ with probability one. For a real number $M$, consider $f_M(x)=M x(x-1)$.

At both training features, $f_M(0)=0$ and $f_M(1)=0$. Thus $R_2(f_M)=0$, for every $M$. The population risk is different:

R(f_M)=\int_0^1 \{M x(x-1)\}^2\,dx =M^2\int_0^1(x^4-2x^3+x^2)\,dx =M^2\left(\frac{1}{5}-\frac{1}{2}+\frac{1}{3}\right) =\frac{M^2}{30}.

For example, $M=6$ gives population risk $36/30=1.2$, despite zero training risk. Increasing $|M|$ makes the risk arbitrarily large without changing the fit at either observed feature.

Worked Example: An Unobserved Group in a Discrete Population

Let $P(X=0)=0.9$, $P(X=1)=0.1$, and let $Y=0$ at both feature values. Suppose the observed training sample contains $n=4$ pairs, all equal to $(0,0)$. This event has probability $0.9^4=0.6561$ under the stated population. Define $f_M(0)=0$ and $f_M(1)=M$.

Every observed response is fitted exactly, so $R_4(f_M)=0$. For a fresh pair, the squared error is zero when $X=0$ and $M^2$ when $X=1$. Therefore

$$ R(f_M)=0.9(0-0)^2+0.1(0-M)^2=0.1M^2. $$

At $M=5$, the population risk is $0.1(25)=2.5$. The risk arises entirely from a feature value absent from this particular training sample. Interpolation says nothing about the predictor's value there unless the model or procedure imposes additional restrictions.

Worked Example: Exact Fit With Zero Population Risk

Now let $X$ be equally likely to be 0 or 1, and set $Y=X$ deterministically. Suppose the training sample contains both possible feature values, with responses $(0,0)$ and $(1,1)$. The predictor $f(x)=x$ interpolates both observations. It also agrees with the response at every feature value in the population:

R_2(f)=\frac{(0-f(0))^2+(1-f(1))^2}{2}=0, \qquad R(f)=\frac{1}{2}(0-f(0))^2+\frac{1}{2}(1-f(1))^2=0.

This example is not in conflict with the first two. A perfect training fit can accompany zero population risk when the predictor is correct throughout the population's support. The point is that interpolation alone does not establish that agreement.

What Population Risk Measures Under Squared Loss

Population prediction risk is expected squared error for a fresh response. It includes both error in predicting the conditional mean and variation in the response around that mean. To state this precisely, let $m(X)=E[Y\mid X]$, and assume $E[Y^2]<\infty$ and $E[f(X)^2]<\infty$. For a fixed predictor $f$, the conditional mean is well defined and square-integrable.

Theorem: Under these moment conditions, $$ R(f)=E[\operatorname{Var}(Y\mid X)]+E[(m(X)-f(X))^2]. $$ The first term is the mean conditional response variance; the second is the squared prediction error relative to the conditional mean.

To prove the identity, write $Y-f(X)=(Y-m(X))+(m(X)-f(X))$, square, and take expectations. The two square terms give the displayed terms because $E[(Y-m(X))^2\mid X]=\operatorname{Var}(Y\mid X)$. For the cross term, $m(X)-f(X)$ is a function of $X$, so

E[(Y-m(X))(m(X)-f(X))] =E\!\left[(m(X)-f(X))E[Y-m(X)\mid X]\right]=0,

since $E[Y-m(X)\mid X]=0$. The expectations and expansion are justified by square integrability and the Cauchy–Schwarz inequality. Adding the terms proves the identity. $\square$

Thus, even the best possible predictor under squared loss generally has positive risk when $Y$ varies conditionally on $X$. In the special case $Y=0$ deterministically, used in the first two examples, $m(X)=0$ and the conditional variance is zero, so the risk is exactly $E[f(X)^2]$. A zero empirical risk in that setting still does not force $f(X)=0$ at unobserved feature values.

Why Interpolation Alone Places No Upper Bound on Risk

The continuous-feature example illustrates a general fact: without restrictions on the predictor, exact agreement at finitely many points does not control its behavior elsewhere. The following result makes that statement precise for one-dimensional regression. It concerns a fixed dataset and does not assert that every fitted procedure will produce a high-risk interpolator.

Theorem: Fix distinct real training features $x_1,\ldots,x_n$ and responses $y_1,\ldots,y_n$. Suppose the population feature $X$ has bounded support and a nonatomic distribution, and $E[Y^2]<\infty$. Among polynomial predictors that interpolate these observations, population squared risk is unbounded above.

Proof. There is a polynomial $p$ of degree at most $n-1$ satisfying $p(x_i)=y_i$ for all $i$, by polynomial interpolation at distinct points. Set $g(x)=\prod_{i=1}^n(x-x_i)$, and for each real $t$ define $f_t(x)=p(x)+t g(x)$. Since $g(x_i)=0$, every $f_t$ interpolates the data.

On the bounded support of $X$, the polynomials $p$ and $g$ are bounded. Also, $g(X)\ne0$ with probability one: its only real roots are the finitely many $x_i$, each of which has probability zero under a nonatomic distribution. Consequently $A=E[g(X)^2]$ is finite and strictly positive. By square integrability of $Y$ and boundedness of $p(X)$ and $g(X)$, the quantities $C=E[(Y-p(X))^2]$ and $B=E[(Y-p(X))g(X)]$ are finite. Expanding gives

$$ R(f_t)=E[(Y-p(X)-t g(X))^2]=C-2tB+t^2A. $$

Since $A>0$, the quadratic expression tends to infinity as $|t|$ tends to infinity. Each predictor in this sequence has empirical risk zero, so the population risks of interpolators are unbounded above. $\square$

Interpretation and a Common Pitfall

An interpolator has training risk zero, but its prediction risk is a separate quantity determined by its behavior on new observations. For a data-dependent estimator, the conditional population risk can vary with the realized sample. Assessing expected performance across training samples requires an additional expectation over those samples; it is not the same operation as evaluating training error.

Wrong reasoning: “The fitted model has zero training error, so its prediction risk must be zero.”
Correction: Zero training error establishes equality only at the observed training pairs. Population risk averages squared error over a fresh draw from $P$, including feature values not observed in training.

Interpolation is a description of fit, not by itself a guarantee of generalization or a guarantee of failure. The examples show zero risk is possible, while the theorem shows that unrestricted interpolators can also have arbitrarily large risk. Any further conclusion requires information about the predictor, the population distribution, or both.

Check Your Understanding

Use the distinction between sample averages and population expectations to answer the following.

For squared loss, why does $R_n(f)=0$ imply that every observed response is fitted exactly?
In the continuous-feature example, calculate the risk at $M=3$. Does changing $M$ change either fitted training value?
In the discrete example, what is the population risk if $M=0$, and what does this say about interpolation by itself?
State the two terms in the squared-loss risk decomposition and explain which one remains when $Y$ is deterministic given $X$.
Why does the polynomial-interpolation theorem require the feature distribution to be nonatomic?