Why Training Error Understates Prediction Error

Why a Small Residual Error Is Not the Whole Story

A regression model is fitted to observed outcomes, so its predictions are shaped by the very data on which its residuals are calculated. This makes the residual error useful for describing how closely the fitted model matches those observations, but it does not by itself tell us how well the model will predict outcomes it has not seen.

To assess prediction, compare performance on the data used for fitting with performance on separate, held-out observations. These quantities can differ substantially. The distinction matters because a model can fit quirks of its training sample as well as any underlying relationship. A good fit to those quirks need not translate into accurate predictions for new cases.

Definition: For training observations \((x_i,y_i)\), let \(\hat{y}_i\) be the fitted prediction and \(e_i=y_i-\hat{y}_i\) the residual. The training mean squared error is the average squared residual on the observations used to fit the model. For held-out observations \((x_j^{\mathrm{test}},y_j^{\mathrm{test}})\), the held-out mean squared prediction error is the average squared difference between their outcomes and predictions from the already-fitted model. Both are measured in squared outcome units.

\mathrm{MSE}_{\mathrm{train}}=\frac{1}{n_{\mathrm{train}}}\sum_{i=1}^{n_{\mathrm{train}}}(y_i-\hat{y}_i)^2, \qquad \mathrm{MSE}_{\mathrm{test}}=\frac{1}{n_{\mathrm{test}}}\sum_{j=1}^{n_{\mathrm{test}}}(y_j^{\mathrm{test}}-\hat{y}_j^{\mathrm{test}})^2

The training predictions in the first expression come from a model fitted using those training outcomes. In the second expression, the model is held fixed: the held-out outcomes must not have been used to fit it. A held-out evaluation is informative about prediction on new cases when those cases reasonably represent the same setting and outcome process. A small held-out sample can still give a noisy estimate, and it may happen to produce a smaller error than the training data.

Calculating Both Errors for a Fitted Line

Worked Example: Training and Held-Out Error for a Linear Regression

Suppose a hypothetical tutor records study hours \(x\) and quiz score \(y\), measured in points. Four students form the training set: \((1,2),(2,3),(3,5),(4,4)\). Two other students are held out: \((5,7)\) and \((6,5)\). Fit a straight line by least squares using only the four training observations, then calculate each error measure.

The candidate model is a straight-line relationship between study hours and the mean quiz score. For this calculation, the line is simply the specified prediction rule; its errors can be computed without assuming normally distributed outcomes. For the held-out error to represent future prediction in this example, suppose the held-out students are independent of the training students and come from the same relevant setting. The hypothetical data alone cannot establish that condition.

The training means are \(\bar{x}=2.5\) hours and \(\bar{y}=3.5\) points. The least-squares slope is the sum of products of deviations divided by the sum of squared \(x\)-deviations:

b_1=\frac{(-1.5)(-1.5)+(-0.5)(-0.5)+(0.5)(1.5)+(1.5)(0.5)} {(-1.5)^2+(-0.5)^2+(0.5)^2+(1.5)^2} =\frac{4}{5}=0.8

The intercept is \(b_0=\bar{y}-b_1\bar{x}=3.5-(0.8)(2.5)=1.5\). The fitted line therefore predicts quiz score \(\hat{y}=1.5+0.8x\). Applying it to the training observations gives predictions \(2.3,3.1,3.9,4.7\) points and residuals \(-0.3,-0.1,1.1,-0.7\) points. Thus:

\mathrm{MSE}_{\mathrm{train}}= \frac{(-0.3)^2+(-0.1)^2+(1.1)^2+(-0.7)^2}{4} =\frac{1.80}{4}=0.450 \ \text{points}^2

For the held-out students, the model predicts \(1.5+(0.8)(5)=5.5\) points and \(1.5+(0.8)(6)=6.3\) points. The prediction errors, defined as observed score minus predicted score, are \(7-5.5=1.5\) points and \(5-6.3=-1.3\) points. Therefore:

\mathrm{MSE}_{\mathrm{test}}= \frac{(1.5)^2+(-1.3)^2}{2} =\frac{3.94}{2}=1.97 \ \text{points}^2

The held-out error is larger than the training error in this particular split: \(1.97\) rather than \(0.450\) points squared. This difference is not a universal multiplier and does not prove that every new student’s prediction will be poor. It shows that the close fit to the four training scores did not guarantee equally close predictions for these two held-out scores.

Why Fitting Creates Optimism

Least squares chooses model coefficients to make squared residuals on the fitting data as small as possible. A parameter value that happens to accommodate a random high or low outcome can improve the training fit, even if that random feature does not recur in new observations. Held-out outcomes did not help choose the coefficients, so their errors include the consequences of that fitting process.

A simple comparison makes the optimization point precise. Suppose a straight-line model includes an intercept and a slope, while a constant-only model predicts the same sample mean at every \(x\). The straight-line model can reproduce any constant-only prediction by setting its slope to zero. Since least squares chooses the line with the smallest training sum of squared errors among all such lines, its training error cannot be greater than the constant-only model’s training error. This guarantee concerns the data used in fitting only. It says nothing that forces the line’s held-out error to be smaller.

\min_{b_0,b_1}\sum_{i=1}^{n}(y_i-b_0-b_1x_i)^2 \ \leq\ \min_{b_0}\sum_{i=1}^{n}(y_i-b_0)^2

The inequality follows because every constant prediction \(b_0\) on the right is also available to the line on the left by taking \(b_1=0\). A more flexible model may use its additional choices to follow random variation in the training outcomes. That can reduce training error while failing to improve, or even worsening, prediction on new outcomes.

Worked Example: A Quadratic That Interpolates Its Training Data

Consider hypothetical training observations of a predictor \(x\) and outcome \(y\), both measured in suitable units: \((0,0),(1,1),(2,0)\). Fit a quadratic model \( \hat{y}=a+bx+cx^2\). Suppose a fourth, independent observation from the same setting is held out at \(x=0.5\), with observed outcome \(y=-1\). This example assumes the quadratic form was specified before examining the held-out outcome; the held-out point lies within the training range of \(x\).

The quadratic \( \hat{y}=2x-x^2\) passes exactly through all three training points: it predicts \(0\) at \(x=0\), \(1\) at \(x=1\), and \(0\) at \(x=2\). Its three training residuals are therefore all zero, so \(\mathrm{MSE}_{\mathrm{train}}=0\) outcome-units squared.

At \(x=0.5\), the prediction is \(2(0.5)-(0.5)^2=0.75\). The held-out prediction error is \(-1-0.75=-1.75\), and its squared error is \((-1.75)^2=3.0625\) outcome-units squared. The model has zero training error but nonzero held-out error. With just one held-out case, that error is not a stable estimate of average future performance; the calculation illustrates that a perfect fit to training observations does not imply perfect prediction.

A Short Derivation of Expected Optimism

The difference can be shown exactly for the simplest regression model, an intercept-only model. Suppose \(Y_1,\ldots,Y_n\) are independent outcomes from the same distribution, each with mean \(\mu\) and variance \(\sigma^2\). The fitted intercept is the training mean \(\bar{Y}\). No normality assumption is needed; independence and a finite variance suffice for the expectations below.

The identity below follows by expanding both sides and using \(\sum_{i=1}^n(Y_i-\bar{Y})=0\). Also, \(E[(\bar{Y}-\mu)^2]=\mathrm{Var}(\bar{Y})=\sigma^2/n\).

\begin{aligned} \sum_{i=1}^{n}(Y_i-\bar{Y})^2 &=\sum_{i=1}^{n}(Y_i-\mu)^2-n(\bar{Y}-\mu)^2,\\ E\left[\frac{1}{n}\sum_{i=1}^{n}(Y_i-\bar{Y})^2\right] &=\frac{1}{n}\left(n\sigma^2-n\frac{\sigma^2}{n}\right) =\frac{n-1}{n}\sigma^2. \end{aligned}

Now let \(Y_{\mathrm{new}}\) be an independent outcome from the same distribution. It is independent of \(\bar{Y}\), so the expected squared prediction error for this new case is the sum of their variances around \(\mu\):

E[(Y_{\mathrm{new}}-\bar{Y})^2] =\mathrm{Var}(Y_{\mathrm{new}}-\bar{Y}) =\sigma^2+\frac{\sigma^2}{n}.

Subtracting the expected training mean squared error from this expected new-case error gives \(2\sigma^2/n\), which is positive when \(\sigma^2>0\). In this particular model and setting, training error is optimistic on average, even though a particular held-out sample can still happen to have lower error than the training sample.

Worked Example: Putting the Intercept-Only Result Into Numbers

Suppose a hypothetical process produces independent outcomes with a constant mean and variance \(\sigma^2=4\) points squared. An intercept-only model is fitted to \(n=4\) training outcomes by predicting their sample mean. The assumptions for the derivation are that the outcomes are independent, share this same mean and variance, and have finite variance. Normality is not required.

The expected training mean squared error is \(((n-1)/n)\sigma^2=((4-1)/4)(4)=3\) points squared. For one independent new outcome, the expected squared prediction error is \((1+1/n)\sigma^2=(1+1/4)(4)=5\) points squared. The expected difference is \(5-3=2\) points squared, also given by \(2\sigma^2/n=2(4)/4=2\). These are averages over repeated samples under the stated model, not guarantees about the error in any one split.

When the Gap Can Be Large—and What It Does Not Mean

The training–held-out gap can be especially noticeable when a model has many ways to adjust itself relative to the amount of training information. A highly flexible curve can follow individual training points; a model fitted on a small sample can be strongly influenced by a few observations; and fitting many candidate specifications and retaining one that looks best on those same data can take advantage of chance patterns. In each case, the training outcomes have helped shape the fitted predictions.

A large gap can also arise when new cases differ from the training cases in important ways. For instance, predicting far outside the observed range of a predictor can be unreliable even when the model’s training residuals are small. This is a change in the prediction setting, not just the optimism caused by fitting on the training outcomes. Held-out data are most useful for assessing future performance when they reflect the cases to which the prediction procedure will be applied.

A gap is not itself proof that the regression model is wrong, and a small gap is not proof that the model predicts well. Both errors might be large, or both might be small. The absolute held-out error and the intended prediction setting matter, as does the amount of uncertainty in an estimate based on a limited number of held-out cases.

Common Mistake

Wrong reasoning: “The model with the smallest training error must be the best predictor, because it fits the observed data most closely.”
Correction: Training error measures fit on outcomes used to estimate the model. A more flexible model can lower that error by following sample-specific noise. Compare prediction error on observations not used for fitting to assess how well the fitted procedure predicts new cases; do not infer better generalization from a training-error decrease alone.
Another mistake: “Held-out error must be greater than training error.” The worked calculations show why it can be greater, but the actual ordering can go either way because held-out outcomes vary. The optimism claim concerns average performance under suitable conditions, not a guaranteed inequality for every split.

Key takeaway: Residual error on fitting data describes in-sample fit, not independent prediction performance. Since the model was adjusted using those outcomes, training error can be optimistic; a held-out evaluation gives a separate, though still uncertain, check on prediction.

Check Your Understanding

Use the definitions and reasoning above to answer these questions.

In your own words, distinguish a training residual from a held-out prediction error. Which outcomes were used to fit the model in each calculation?
Why can least-squares training error never increase when a straight-line model is allowed to use a slope instead of being restricted to a constant prediction?
For the intercept-only model in the derivation, what assumptions make the expected training and new-case errors comparable?
Can a held-out error be smaller than the training error for one particular split? Explain why this would not contradict average optimism.
Name two features of a modelling situation that can make training error particularly optimistic, and explain the role of fitting in each.