What the Training Set Can and Cannot Tell You

Introduction

A model can predict its training observations extremely well and still make poor predictions for new cases. The distinction matters whenever a model is fitted and its performance is then summarized using the same data: the reported error describes the cases the model has already seen, not automatically the cases it has not.

Training error is still useful. It tells us how closely the fitted model matches the data used to fit it, and can reveal obvious failures to capture those observations. The essential limit is that the model was chosen using those observations. Its fit can reflect both genuine structure and random features peculiar to the training sample.

Definition: Given training observations $(x_i,y_i)$, $i=1,\ldots,n$, a fitted prediction rule $\hat f$, and a loss function $L(y,\hat y)$, the training error is

\widehat R_{\mathrm{train}} = \frac{1}{n}\sum_{i=1}^{n}L\bigl(y_i,\hat f(x_i)\bigr).

It is the average loss on the observations used to fit $\hat f$. By contrast, the new-data prediction risk, conditional on the fitted rule, is the expected loss on a new observation from the target population:

R(\hat f) = E\!\left[L\bigl(Y_{\mathrm{new}},\hat f(X_{\mathrm{new}})\bigr)\mid \text{training data}\right].

The two quantities use the same loss but refer to different observations.

What Training Error Directly Describes

The training error is a summary of the model's performance on the particular data used to fit it. Its interpretation depends on the loss. With squared-error loss, it is the average squared difference between observed responses and fitted predictions. With absolute-error loss, it is the average absolute difference. For classification with zero-one loss, it is the proportion of training labels predicted incorrectly.

A small training error supports a precise, limited statement: the fitted rule has low average loss on these observed training cases under the chosen loss. It does not by itself establish that the rule will have low loss on future cases. In particular, a flexible fitting procedure may adapt to noise in the training observations as well as to the underlying relationship.

Worked Example: Calculating Training Error for Claim-Cost Predictions

A hypothetical insurer fits a model to five training records. Claim costs and predictions are recorded in thousands of dollars. Using these training cases only, calculate the mean absolute error, mean squared error, and root mean squared error. State what the calculations do and do not establish.

Record	Observed cost ($000s)	Fitted prediction ($000s)	Observed minus predicted ($000s)
1	2	3	-1
2	4	3	1
3	5	6	-1
4	9	8	1
5	10	9	1

Step 1: Find the absolute errors. The absolute errors are $1,1,1,1,1$ thousand dollars. Their sum is $5$, so the training mean absolute error is $5/5=1$ thousand dollars.

Step 2: Find the squared errors. The squared errors are $1,1,1,1,1$, in squared thousands of dollars. Their sum is $5$, so the training mean squared error is $5/5=1$ squared thousand dollars.

Step 3: Take the square root for RMSE. The training root mean squared error is $\sqrt{1}=1$ thousand dollars.

Interpretation and check. For these five fitted records, all three summaries give an error magnitude of $1$ thousand dollars (MSE is $1$ in squared-thousand-dollar units). This verifies the arithmetic against the table. These values describe the training records; without evidence from new observations, they do not establish the model's prediction error on future claim costs. The common mistake is to label a training summary as a measured future-data error.

Why Fitting Can Make Training Error Optimistic

When the same responses are used both to choose a prediction rule and to calculate its error, the fitting procedure has had an opportunity to reduce error on those specific responses. This does not mean training error is always smaller than every future-data error for every individual sample. It means that, on average under common sampling assumptions, fitting to the training sample can make its own error look better than the error on a fresh observation.

The following simple model makes that effect explicit. Suppose $Y_1,\ldots,Y_n$ are independent observations with common mean $\mu$ and variance $\sigma^2$. Fit a constant prediction equal to the training sample mean $\bar Y$. Use squared-error loss. The model is deliberately simple: the point is to isolate the effect of estimating the prediction from the same responses on which fit is measured.

Derivation: The fitted constant is $\bar Y$. The expected training mean squared error is

E\!\left[\frac{1}{n}\sum_{i=1}^{n}(Y_i-\bar Y)^2\right] = \frac{n-1}{n}\sigma^2.

To see why, use the identity $\sum_i(Y_i-\bar Y)^2=\sum_i(Y_i-\mu)^2-n(\bar Y-\mu)^2$. The expected first term is $n\sigma^2$, and $E[n(\bar Y-\mu)^2]=n\operatorname{Var}(\bar Y)=n(\sigma^2/n)=\sigma^2$. Dividing their difference, $(n-1)\sigma^2$, by $n$ gives the result.

For a new independent response $Y_{\mathrm{new}}$ with the same mean and variance, the prediction error is $Y_{\mathrm{new}}-\bar Y$. Independence gives

E[(Y_{\mathrm{new}}-\bar Y)^2] = \operatorname{Var}(Y_{\mathrm{new}})+\operatorname{Var}(\bar Y) = \sigma^2+\frac{\sigma^2}{n}.

Both quantities are expectations over the training sample (and, for new-data error, the independent new response). Their difference is $2\sigma^2/n$, which is positive when $\sigma^2>0$. Thus, in this model, the expected training error is lower than the expected prediction error for a new response.

Worked Example: Quantifying the Training-to-Prediction Gap

Assume a hypothetical set of four policy-duration observations follows the constant-mean model just described: the observations are independent, have common mean $\mu$, and have variance $\sigma^2=9$ squared units. A constant prediction is fitted using the sample mean. Calculate the expected training MSE, expected squared prediction error for one new observation, and their difference.

Step 1: Expected training MSE. Here $n=4$, so \[ \frac{n-1}{n}\sigma^2 = \frac{4-1}{4}(9) = \frac{3}{4}(9) = 6.75. \]

Step 2: Expected new-observation squared prediction error. The variance of the sample mean is $\sigma^2/n=9/4=2.25$. Therefore, \[ \sigma^2+\frac{\sigma^2}{n} = 9+2.25 = 11.25. \]

Step 3: Compare. The expected difference is $11.25-6.75=4.50$. The general expression gives the same result: $2\sigma^2/n=2(9)/4=4.50$.

Check and scope. The two calculations agree on a gap of $4.50$ squared units. These are expected errors under the stated model, not guaranteed values for any particular sample. This result is not a universal adjustment that can be subtracted from or added to every training error: the gap depends on the model, fitting procedure, sample, and loss. A common trap is to report the formula as though it applied to any fitted model; it was derived here for a fitted constant and squared-error loss under the stated independence and variance assumptions.

Low Training Error Is Not a Generalization Guarantee

A model that can closely adapt to individual training observations may achieve a very low training error. If it has enough flexibility, it may even reproduce the training responses exactly. That result confirms a close fit to the data used in fitting; it does not tell us whether the details it learned will recur in new observations.

The following comparison shows why a near-perfect training summary is not sufficient evidence of future performance. The models and values are hypothetical; no separate future outcomes are supplied.

Worked Example: Exact Training Fit Versus Evidence About New Cases

A hypothetical analyst predicts renewal expenses for four training records, in thousands of dollars. Model A is a relatively constrained rule; Model B is a more flexible rule fitted to the same four records. Calculate each model's training MAE and MSE, then state the strongest conclusion justified by these results.

Record	Observed expense	Model A prediction	Model B prediction
1	2	3	2
2	5	4	5
3	7	8	7
4	10	9	10

Step 1: Model A errors. The absolute errors are $1,1,1,1$, giving a total of $4$ and a training MAE of $4/4=1$ thousand dollars. The squared errors are also $1,1,1,1$, so the training MSE is $4/4=1$ squared thousand dollars.

Step 2: Model B errors. All four predictions equal the observed expenses. Each absolute and squared error is $0$, so its training MAE and MSE are both $0$.

Step 3: Interpret the comparison. Model B has lower training error on these four records. The data supplied do not establish that it will have lower error on new cases: Model B's exact fit may reflect recurring signal, adaptation to sample-specific noise, or both. There are no new outcomes here with which to determine that. The common mistake is to infer “best on future cases” from “best on the training cases.” The calculation supports only the latter statement.

The training set can also help flag a model that plainly fails to represent the observed cases. For example, large training errors or systematic residual patterns may prompt a review of the model and data. But a model that passes this in-sample check has not thereby passed a check on new observations. Training fit is informative about the observed sample; its evidential reach should not be overstated.

Assumptions Behind the Interpretation

It is important to identify what kind of comparison is being made. If a prediction rule is fixed independently of an evaluation sample, and the observations in that sample are independent draws from the target population, its average loss is an unbiased estimate of that fixed rule's expected loss. The reasoning is direct: each loss has the same expected value $R(f)$, so the average does too.

E\!\left[\frac{1}{m}\sum_{j=1}^{m}L(Y_j,f(X_j))\right] = \frac{1}{m}\sum_{j=1}^{m}E[L(Y_j,f(X_j))] = R(f).

That argument requires that the rule $f$ was not fitted or selected using those evaluation observations. When the rule is fitted on the same observations used in the average, the losses are affected by the fitting process. When a model is chosen after comparing alternatives on the same observations, the reported minimum training error is affected by that choice as well.

The independent, same-distribution condition also matters. Even an evaluation sample that was not used for fitting may provide misleading evidence about the intended population if its cases are dependent, selected in a systematically different way, or drawn from a changed population. Training error alone cannot diagnose all such problems, and a low value does not eliminate them.

Common Mistakes

Calling training error “test error.” Training error is calculated on data used for fitting. It is not a direct measurement of loss on new cases.
Treating zero training error as proof of a good model. It proves only that the fitted rule matches the training responses exactly under the stated loss.
Assuming training error must be lower on every particular sample. The optimism argument is generally about expected behavior under assumptions; realized errors can vary.
Ignoring the loss and its units. MSE has squared response units; MAE has the response's original units. Neither should be described without identifying the loss used.
Applying a universal optimism correction. The constant-mean derivation above depends on its specific model, assumptions, and squared-error loss; it is not a formula for every learning procedure.

Key takeaway: Training error measures fit to the observations used to fit the model. It can reveal how well the model represents those observations, but a very low value alone is not evidence that the same performance will hold for new data.

Check Your Understanding

For each problem, distinguish what can be calculated from the training cases from what can be concluded about new cases.

A model's fitted predictions for four training observations are $4, 6, 7, 11$, and the observed values are $3, 7, 7, 9$. Calculate the training MAE and MSE. (Answer: MAE $=1.00$; MSE $=1.50$, in squared response units.)
A classifier correctly labels 27 of its 30 training records. What is its training misclassification rate, and does that number alone establish its error rate on new records? (Answer: Training error $=3/30=0.1000$, or $10\%$; no, it does not by itself establish new-record error.)
Under the independent common-mean model with squared-error loss, a constant is fitted to $n=9$ observations with variance $\sigma^2=4$. Find expected training MSE and expected squared prediction error for one new observation. (Answer: $32/9\approx3.5556$ and $40/9\approx4.4444$, respectively.)
A flexible model has zero training MSE, while a constrained model has positive training MSE. No outcomes from new observations are provided. What comparison is justified? (Answer: The flexible model has lower training MSE; the data given do not establish which model has lower new-data prediction error.)
A prediction rule is fixed independently of 20 independent evaluation observations from the target population. The average squared loss on those observations is $18$. What does this average estimate, and what condition makes its expectation equal to the rule's risk? (Answer: It estimates the fixed rule's expected squared prediction loss; with independent evaluation observations from the target population and a rule not fitted using them, its expected value equals that risk.)

What You'll Learn