Common Misreadings of Generalization Performance

Why Performance Numbers Are Easy to Misread

A low error is meaningful only when you know which observations produced it and how the model was chosen. A model can fit its training data very closely yet predict new observations poorly. A more complex model can improve training fit while worsening validation performance. And a test score that influenced which model was selected is no longer a clean final evaluation.

These distinctions matter because the goal of supervised learning is usually prediction for observations not used to fit or select the model. The training score describes fit to the data used for estimation. The validation score helps compare candidate models or choices. A test score is intended to evaluate the final selected procedure on data kept separate from those decisions. Those roles are not interchangeable.

Definition: The generalization risk of a fixed prediction rule is its expected loss on a new observation drawn from the target population. For a predictor $ \hat f $ and loss function $ L $, this is $ R(\hat f)=E[L(Y,\hat f(X))] $. The training error is the average loss on the observations used to fit the rule; it is not generally an unbiased estimate of generalization risk because the rule was fitted to those observations.

The expectation in generalization risk is over new observations from the population to which predictions are intended to apply. If that population changes, performance on an earlier sample may not describe the new target. Even when the population is stable, a finite validation or test score varies from sample to sample.

Training Fit Is Not Predictive Quality

A learning procedure chooses a model using the training data. Consequently, the fitted model is selected in part to perform well on that particular sample. Its training error can be smaller than its error on new observations, especially when the model is flexible relative to the amount of data. Training error is useful for understanding fit, but it is not by itself a basis for claiming predictive accuracy.

For example, under squared-error loss, training mean squared error is the average of squared differences between observed outcomes and fitted predictions on the training observations. Validation mean squared error is calculated in the same way on a separate validation sample. The loss formula is the same; the observations and the interpretation differ.

Worked Example: A Better Training Fit but Worse Validation Fit

A hypothetical insurer is comparing two models for predicting a service cost, measured in thousands of dollars. Squared-error loss is the prediction goal. On three training cases, actual costs are $10, 12,$ and $14$. Model A predicts $11, 12,$ and $13$; Model B predicts $10, 12,$ and $14$. On two separate validation cases, actual costs are $9$ and $13$. Model A predicts $10$ and $12$; Model B predicts $8$ and $15$. Calculate both models’ training and validation mean squared errors and state which model the validation results favor.

Method. For each data set, calculate each squared prediction error, sum them, and divide by the number of observations in that data set.

Model A, training. Its errors are $10-11=-1$, $12-12=0$, and $14-13=1$. Their squares are $1,0,1$, so the sum of squared errors is $1+0+1=2$. The training mean squared error is $2/3=0.6667$, rounded to four decimal places.

Model B, training. Its three predictions match the actual values, so its squared errors are $0,0,0$. The sum is $0$, and the training mean squared error is $0/3=0$.

Model A, validation. Its errors are $9-10=-1$ and $13-12=1$, with squares $1$ and $1$. The validation mean squared error is $(1+1)/2=1$.

Model B, validation. Its errors are $9-8=1$ and $13-15=-2$, with squares $1$ and $4$. The validation mean squared error is $(1+4)/2=2.5$.

The validation results favor Model A: its mean squared error is $1$, compared with $2.5$ for Model B. Model B fits the training cases better but predicts these validation cases less accurately. The comparison is based on only two validation observations, so it is uncertain; it does not establish that Model A will always predict better. The mistake to avoid is declaring Model B the better predictive model solely because its training error is smaller.

More Complexity Does Not Guarantee Better Generalization

As a model’s flexibility increases, its training fit often improves because the model can accommodate more patterns in the training sample. That improvement does not guarantee a reduction in generalization risk. A flexible model may also respond to chance variation in the training data, producing predictions that do not carry over to new observations.

This is why complexity should be judged using data not used to fit the candidate models. A validation set can compare specified candidate choices. If validation results tie or differ only slightly, the data may not justify claiming a clear winner. A simpler candidate may be a reasonable choice when its measured performance is comparable, but that is a decision under uncertainty, not proof that simplicity always predicts better.

Worked Example: Reading a Complexity Table Carefully

A hypothetical screening program compares four classification models of increasing complexity. Each model is fitted using 40 training cases and evaluated on 60 validation cases. The number of classification errors is shown below. Use the validation error rate to identify the best measured performance, and explain why the most complex model should not be selected automatically.

Complexity level	Training errors out of 40	Validation errors out of 60
1	8	12
2	5	9
3	3	9
4	1	15

Step 1: Calculate training error rates. At level 1 the rate is $8/40=0.20$. At level 2 it is $5/40=0.125$. At level 3 it is $3/40=0.075$. At level 4 it is $1/40=0.025$. Training error decreases at every step.

Step 2: Calculate validation error rates. At level 1 the rate is $12/60=0.20$. At level 2 it is $9/60=0.15$. At level 3 it is $9/60=0.15$. At level 4 it is $15/60=0.25$.

Step 3: Compare candidates by the stated prediction goal. Levels 2 and 3 tie for the lowest observed validation error rate, $0.15$. Level 4 has the lowest training error but the highest validation error rate, $0.25$. Thus, the validation data do not favor level 4. If a choice between levels 2 and 3 is required, preferring level 2 is a defensible tie-breaking choice because it is less complex and has the same measured validation error.

Check. The rates use the correct denominators: training errors are divided by 40, while validation errors are divided by 60. The common mistake is to infer that increasing complexity must improve prediction because training errors fall. Here, validation performance improves at first, ties, and then worsens.

Why Repeated Test-Set Use Creates Optimism

A test set can provide a useful final performance estimate if it is kept out of model fitting and selection until the procedure is settled. But the label “test set” does not protect it from bias if its scores are repeatedly examined and used to choose a model, features, thresholds, or other analysis decisions. Once those choices respond to test results, the selected model is partly tailored to that test sample.

The source of the problem is selection. Even if each candidate’s test score is an unbiased estimate of its own risk before comparison, choosing the candidate with the smallest observed score tends to favor a score that benefited from favorable sampling noise. The selected score can therefore be too low, on average, compared with the selected model’s true risk.

Justification: Suppose two fixed candidates have the same true risk $R$, and their test-score errors are $ \varepsilon_1 $ and $ \varepsilon_2 $. The observed scores are $R+\varepsilon_1$ and $R+\varepsilon_2$. If the errors are centered and exchangeable, the expected smaller score is $$ E\left[\min(R+\varepsilon_1,R+\varepsilon_2)\right] =R-\frac{1}{2}E\left[|\varepsilon_1-\varepsilon_2|\right]. $$ This follows from $ \min(a,b)=(a+b-|a-b|)/2 $: take expectations, use $E[\varepsilon_1+\varepsilon_2]=0$, and obtain the displayed expression. Unless the two score errors are identical with probability one, the expected selected score is below $R$. The argument illustrates selection optimism; it is not a formula for the bias in every model-selection setting.

Worked Example: Selecting the Lowest of Repeated Test Scores

For illustration, suppose four candidate models actually have the same generalization error rate, $0.20$. Their reported test error rates are $0.16, 0.19, 0.21,$ and $0.24$. A developer repeatedly compares these test results and chooses the model reporting $0.16$. Explain the resulting misreading, then compare it with a fresh final evaluation that reports $0.21$.

Step 1: Identify the selected score. Among the four reported values, the minimum is $0.16$. The corresponding model is selected.

Step 2: Compare with the stipulated true risk. For this illustrative scenario, that model’s true error rate is $0.20$. Its selected test score is lower by $0.20-0.16=0.04$, or four percentage points. The difference represents favorable test-sample variation in this constructed example; in a real application, the true risk is not known.

Step 3: Interpret the fresh evaluation. If a separate, previously unused test sample gives the final model an error rate of $0.21$, that result is the appropriate final test estimate, subject to ordinary sampling uncertainty and the assumption that the new sample represents the target population. It does not prove that the true risk is exactly $0.21$, but it was not used to select among the four candidates.

The mistake is to report $0.16$ as though it were an untouched final test estimate. Since the candidate was selected partly because it had the lowest score on that repeatedly consulted set, the score is selection-optimistic. The correction is to reserve new data for final evaluation, or to use a resampling design appropriate to the analysis when a truly separate test sample is unavailable.

How to Read Performance Evidence

A careful interpretation identifies the data split, the loss, the comparison being made, and the limitations of the observed result. A score is not meaningful without its context. In particular, do not mix up these two statements:

“This model has a smaller training error.” This describes fit on the data used to estimate the model.
“This model is expected to predict new observations better.” This is a claim about generalization and needs suitable out-of-sample evidence.

The second statement is stronger. A validation comparison can support model selection, but validation results themselves are subject to sampling variation and can become optimistic if repeatedly used to tune many choices. A final test set is meant to evaluate the selected procedure after those choices are complete. If the test results prompt further changes, the set has become part of the development process; a later evaluation then needs new untouched data to serve as a final test.

Also check whether the evaluation sample represents the prediction setting. A carefully reserved test set can still give misleading guidance if it differs systematically from the population where predictions will be used. This is a target-population issue, distinct from the selection optimism caused by repeatedly consulting the same test set.