Training Error and Population Error

Why Two Prediction Errors Matter

A model can make small errors on the observations used to fit it and still predict poorly for new observations. The distinction is not merely about which data are reported: it concerns what quantity is being measured. Training error describes performance on the observed sample. Population prediction error describes expected performance on a new draw from the target population, after the fitted model is held fixed.

This distinction matters whenever a model is assessed for future use, including a hypothetical actuarial model that predicts claim costs from policy characteristics. A low error on the records used to fit a model is evidence about those records; by itself, it does not establish that predictions will be accurate for future policies.

Definition: Let $D=\{(X_i,Y_i):i=1,\ldots,n\}$ be the training data, and let $\hat f_D$ be the predictor fitted using $D$. For a specified loss function $L(y,\hat y)$, the empirical training error is $$ \widehat R_{\mathrm{train}}(D)=\frac{1}{n}\sum_{i=1}^{n}L\bigl(Y_i,\hat f_D(X_i)\bigr). $$ The population prediction error, also called the conditional population risk, is $$ R(\hat f_D)=E_{(X,Y)\sim P}\left[L\bigl(Y,\hat f_D(X)\bigr)\mid D\right]. $$ Here $P$ is the target population distribution. In the second expression, $D$ is held fixed, so the fitted predictor is fixed; the expectation is over a new observation drawn from $P$.

For squared-error prediction, $L(y,\hat y)=(y-\hat y)^2$. For classification, a common choice is zero-one loss, which is 1 when the predicted class is wrong and 0 otherwise. The loss must match the prediction task: these definitions compare errors only after the loss has been specified.

The population error is generally unknown because the full target population is not observed. It is a target quantity, not a number automatically revealed by the training data. Training error is directly calculable, but the predictor in that calculation was selected using those same observations. That dependence is central to the difference.

Fitting and Evaluating on the Same Observations

The training procedure chooses $\hat f_D$ partly in response to the values in $D$. If a model can adapt closely to those values, its training error can be small because it has captured the observed sample’s particular features. Some of those features may not recur in new data. Thus, training error measures both how well the fitted predictor performs and how well it has adapted to the sample on which it was fitted.

This does not mean that training error must always be smaller than population error for every realized sample. A particular training sample may happen to be unusually difficult, and its error can exceed the error on new observations. The important point is that there is no general equality, and a small training error alone gives no guarantee of small population error.

Worked Example: A Fitted Constant and a New-Observation Risk

In a hypothetical study, the response $Y$ takes the value 0 or 10 with probability $1/2$ each. There are no predictors, so the model class consists of constant predictions. A training sample contains $0,0,10$, and the fitted constant minimizes squared training error. Calculate the training error and the population prediction error of the fitted model. Use squared-error loss.

Method. For a constant $c$, squared training error is minimized by the sample mean. The fitted value is $$ \hat c=\frac{0+0+10}{3}=\frac{10}{3}. $$

The three training residuals are $0-\frac{10}{3}=-\frac{10}{3}$, $0-\frac{10}{3}=-\frac{10}{3}$, and $10-\frac{10}{3}=\frac{20}{3}$. Their squared values are $\frac{100}{9}$, $\frac{100}{9}$, and $\frac{400}{9}$. Therefore, $$ \widehat R_{\mathrm{train}} =\frac{1}{3}\left(\frac{100}{9}+\frac{100}{9}+\frac{400}{9}\right) =\frac{600}{27} =\frac{200}{9} \approx 22.22. $$

For a new observation, $Y=0$ or $Y=10$, each with probability $1/2$. Holding the fitted value $10/3$ fixed, the two possible squared losses are $$ \left(0-\frac{10}{3}\right)^2=\frac{100}{9}, \qquad \left(10-\frac{10}{3}\right)^2=\frac{400}{9}. $$ Hence the population prediction error is $$ R(\hat c)=\frac{1}{2}\left(\frac{100}{9}+\frac{400}{9}\right) =\frac{250}{9} \approx 27.78. $$

The population error exceeds the training error in this sample. Both values are calculated using the same squared-error loss, but one averages losses on the observed responses and the other averages over the response distribution of a new observation. Check: the new-observation risk is positive and lies between the two possible squared losses, as it must. The common mistake is to treat the observed training average as the population average simply because both use the same formula.

When an Empirical Average Does Estimate Risk

A useful distinction is whether a predictor was fixed independently of the observations used to calculate its error. Suppose $f$ is a predictor specified before observing a sample. If $(X_i,Y_i)$ are independent draws from $P$, and the loss has a finite expectation, each loss $L(Y_i,f(X_i))$ has expectation $R(f)$. Linearity of expectation then gives

E\left[\frac{1}{n}\sum_{i=1}^{n}L(Y_i,f(X_i))\right] =\frac{1}{n}\sum_{i=1}^{n}E[L(Y_i,f(X_i))] =R(f).

This is an expectation over repeated samples; it does not say that the empirical average in any one sample equals the population risk. More importantly, when $f$ is replaced by $\hat f_D$, the predictor depends on the very observations in the training-error average. The fixed-predictor argument no longer establishes that the expected training error equals the expected population error. No universal equality follows without additional assumptions about the fitting procedure and data.

Result: If the test observations $(X_j^{\mathrm{test}},Y_j^{\mathrm{test}})$, $j=1,\ldots,m$, are independent of $D$ and are independent draws from the same target distribution $P$, then, provided the loss has a finite expectation, $$ \widehat R_{\mathrm{test}} =\frac{1}{m}\sum_{j=1}^{m}L\bigl(Y_j^{\mathrm{test}},\hat f_D(X_j^{\mathrm{test}})\bigr) $$ satisfies $$ E[\widehat R_{\mathrm{test}}\mid D]=R(\hat f_D). $$

The independence from $D$ is essential. Conditional on $D$, the fitted predictor is fixed. Independence ensures that each test observation still has distribution $P$ conditional on $D$, so each test loss has conditional expectation $R(\hat f_D)$. Averaging $m$ such losses preserves that conditional expectation. This justifies test error as an estimate of the fitted predictor’s population error; it does not guarantee that one particular test average equals that error.

Worked Example: Training Error, Test Error, and Population Error

A hypothetical population has four equally likely feature categories $A,B,C,D$, and every response equals 1. A training sample contains one observation at $A$ and one at $B$. A fitted predictor returns 1 for categories observed in the training sample and 0 for other categories. Use squared-error loss. Calculate its training error and population prediction error, then evaluate an independent test sample with categories $C,D,A$.

Method. At $A$ and $B$, the prediction is 1 and the response is 1, so both training losses are $(1-1)^2=0$. Thus, $$ \widehat R_{\mathrm{train}}=\frac{0+0}{2}=0. $$

Conditional on this training sample, the predictor is fixed: it predicts 1 at $A,B$ and 0 at $C,D$. At $A,B$, squared loss is 0; at $C,D$, squared loss is $(1-0)^2=1$. Each category has probability $1/4$, so $$ R(\hat f_D)=\frac{1}{4}(0+0+1+1)=\frac{1}{2}=0.5. $$

For the stated test sample $C,D,A$, the losses are $1,1,0$, respectively. Its test error is $$ \widehat R_{\mathrm{test}}=\frac{1+1+0}{3}=\frac{2}{3}\approx 0.6667. $$ This realized test error is not exactly the population error. If the three test observations are independent of $D$ and independently drawn from the four-category population, each has conditional expected loss $0.5$. Consequently, $$ E[\widehat R_{\mathrm{test}}\mid D] =\frac{0.5+0.5+0.5}{3} =0.5, $$ which agrees with the population error. Check: the training error is zero, but half the target population receives a prediction with squared loss 1. The test average differs from 0.5 because it is based on only three random observations.

The common mistake is to infer that a zero training error proves perfect prediction. Here the predictor has memorized the observed categories; its performance on the unobserved categories determines much of its population error.

The Independence Trap

A test observation must be independent of the training data for the conditional-expectation result above to apply. It is not enough that the test observation has the same marginal distribution as a population observation. If a purported test record is actually one of the training records, its value is already known once $D$ is given. In that case, its conditional distribution given $D$ need not be $P$.

Worked Example: Why Reusing a Training Record Is Not Independent Testing

Use the four-category population and fitted predictor from the previous example: $X$ is uniform on $A,B,C,D$, $Y=1$ always, and the predictor returns 1 at training categories $A,B$ and 0 elsewhere. A proposed test record is the training observation at $A$, reused rather than newly sampled. Compare its loss with the population error.

Method. Since $A$ was in the training sample, the fitted predictor returns $\hat f_D(A)=1$. The reused record also has $Y=1$, so its loss is $$ L(1,\hat f_D(A))=(1-1)^2=0. $$ But the conditional population error is still $0.5$: categories $C,D$, with combined probability $1/2$, each receive prediction 0 and have loss 1.

Thus, the reused record reports zero error, while the population error is 0.5. The failure is not a contradiction: the record is not an independent draw conditional on $D$; it is a known training observation. Check: the population calculation weights the four categories equally and gives $(0+0+1+1)/4=0.5$. The common mistake is to check only that the reused record’s marginal category could occur in the population. The required condition is independence from the training data as well as the correct population distribution.

What to Conclude from Each Error

Training error answers, “How much loss did this fitted predictor incur on the observations used to fit it?” Population prediction error answers, “For this fitted predictor, what loss is expected on a new draw from the target population?” An independent test error is a sample-based estimate of the latter, with its own sampling variability.

The three quantities should not be silently interchanged. In particular:

A low training error describes the fitted data and does not, by itself, establish low population error.
A test average from independent target-population observations has conditional expectation equal to population error, but a finite test average can be above or below that error.
A test set that overlaps with, or otherwise depends on, the training data may fail to estimate population error as intended.
The loss used in training and evaluation should represent the prediction objective; changing the loss changes the error being measured.

The practical discipline is to state which observations define the average, which predictor is being evaluated, what loss is used, and whether the evaluation observations are independent of fitting. This makes the reported error interpretable without claiming more than the data support.

Key takeaway: Training error is an empirical loss on data used to fit the predictor. Population prediction error is the expected loss for that fitted predictor on a new target-population observation. A test average estimates the latter in conditional expectation only when its observations are independent of the training data and are drawn independently from the target population.

Check Your Understanding

For each problem, distinguish the error actually calculated from the population quantity of interest.

A constant predictor is fitted to responses $1,0,0,0,0$ by minimizing squared error. The target response is 1 with probability 0.3 and 0 otherwise. Find the fitted constant, its training error, and its population prediction error. (Answer: fitted constant $0.2$; training error $0.16$; population error $0.22$.)
A population has three equally likely categories $A,B,C$, with deterministic responses $0,1,2$, respectively. Training data contain $A$ and $B$. A predictor returns the observed response at those categories and predicts 10 at $C$. Find training error and population error under squared-error loss. (Answer: training error $0$; population error $64/3$, or approximately $21.3333$.)
A fitted predictor is fixed conditional on training data $D$. Three test observations are independent of $D$, are independent draws from the target population, and each test loss has conditional expectation $0.4$. What is the conditional expectation of their mean test loss? Does that force the observed mean to equal $0.4$? (Answer: conditional expectation $0.4$; no, a realized finite-sample mean need not equal it.)
In a population, $X$ is equally likely to be $A$ or $B$, and $Y=1$ always. A fitted predictor memorizes training category $A$, predicting 1 there, and predicts 0 at $B$. Find its population error under zero-one loss. What loss results if the training record at $A$ is reused as the sole “test” record? (Answer: population error $0.5$; reused-record loss $0$.)
Suppose test observations are independent of each other and each has the target population’s marginal distribution, but they are copied from the training data. Is it valid to conclude that their mean test error has conditional expectation equal to $R(\hat f_D)$? State the missing condition. (Answer: no; the test observations must also be independent of $D$.)

What You'll Learn