Reading Training and Validation Results
A model can fit its training observations well and still predict poorly for new observations. Conversely, a model can perform poorly on both the training and validation data. Comparing the two errors helps diagnose these situations, but the comparison is evidence rather than proof: the result depends on the loss being measured, the data split, and how much validation data is available.
The aim is not simply to choose the model with the smallest training error. Training error describes fit to observations used to estimate the model. Validation error describes performance on a separate set not used to fit it, and can help compare candidate models. The validation set must represent the prediction setting of interest, and the same outcome definition, prediction rule, and loss must be used for the models being compared.
For a regression model, a common loss is squared error. If the observed response is \(y_i\) and the prediction is \(\hat y_i\), the squared error is \((y_i-\hat y_i)^2\). Mean squared error (MSE) is the average squared error across the relevant observations. For classification, one possible loss is the fraction of observations classified incorrectly. The loss should be selected to reflect the prediction goal; raw error rates, for example, may not reveal performance on an important minority class.
What the Comparison Can—and Cannot—Show
A simple model may be too restricted to capture important structure in the prediction problem. If it makes substantial errors on both training and validation data, underfitting is one possible explanation. A more flexible model may reduce training error substantially. If its validation error also falls, the additional flexibility may be useful. If its training error falls while validation error rises, the model may be fitting details specific to the training observations that do not carry over to new cases.
The difference between training and validation error is often called the train-validation gap. Its size is meaningful only in context. A gap can reflect overfitting, but it can also be affected by the particular observations in the split, validation-set size, or a mismatch between the training and validation populations. Small gaps do not establish that a model will generalize well: both errors might be large, or the validation set might be unrepresentative.
Comparisons also require like-for-like measurements. If one model is evaluated using MSE and another using absolute error, their reported values cannot be directly ranked. Even when both use the same loss, the loss values must be interpreted in their units. MSE is measured in squared response units, while root mean squared error (RMSE) is in the response's original units.
Worked Example: Comparing Regression Models by Validation MSE
A hypothetical insurer models the annual repair cost, in thousands of dollars, for a type of commercial equipment. Three candidate regression models are evaluated using MSE, in squared thousands of dollars. Their training and validation results are:
| Model | Training MSE | Validation MSE |
|---|---|---|
| Linear | 25 | 29 |
| Cubic | 9 | 24 |
| Degree 8 | 2 | 40 |
Method. Compare the validation MSEs to identify which candidate has the best observed validation performance, then compare each model's training and validation results for diagnostic clues.
Step 1: Compare validation error. The three validation MSEs are 29, 24, and 40. The cubic model has the smallest observed validation MSE, 24. Thus, among these candidates, it performs best on this validation set under the stated loss.
Step 2: Calculate the gaps. Subtract training MSE from validation MSE for each model:
The linear model has relatively high error on both sets compared with the cubic model, which is consistent with possible underfitting. The degree 8 model has very low training MSE but the highest validation MSE and the largest gap; this is evidence consistent with overfitting. The cubic model has the lowest validation MSE, though its gap shows that its training fit is better than its validation performance.
Step 3: Check the interpretation. The MSE values are in squared thousands of dollars. For example, the cubic validation RMSE is \(\sqrt{24}\), approximately 4.899 thousand dollars, while its training RMSE is \(\sqrt{9}=3\) thousand dollars. These are square roots of the stated MSEs, not new evaluation results.
Common mistake and correction. Selecting the degree 8 model because its training MSE of 2 is smallest would rank fit to the training data, not performance on held-out observations. Under the common validation-selection procedure, the cubic model is preferred because its validation MSE, 24, is lowest. This does not establish that the cubic model is best for all future data; an untouched test set or further representative evidence would strengthen the conclusion.
Why Validation Error Estimates Future Performance
The rationale for validation error can be stated precisely. Let the fitted model \(f\) be fixed after training. Suppose the validation observations \(Z_1,\ldots,Z_m\) are independent draws from the target population and are independent of the training data. Let \(L(Z_j,f)\) denote the loss on validation observation \(j\), and suppose its expected value is finite. The target-population risk of this fixed fitted model is \(R(f)=E[L(Z,f)]\), where \(Z\) is a new observation from that population. The average validation loss is
Conditioning on the training data makes \(f\) fixed. Each validation observation then has expected loss \(R(f)\). By linearity of expectation,
Thus, under these conditions, validation error is an unbiased estimate of the risk of a fixed fitted model. A particular validation result can still differ from that risk, especially when \(m\) is small. Further, if the same validation set is repeatedly used to choose among models, the selected model is no longer independent of those validation results. Its minimum observed validation error can be optimistic. An untouched test set is useful for a final assessment after model selection.
Worked Example: Interpreting Classification Error Rates
A hypothetical claims team predicts whether a submitted equipment claim will require specialist review. In the training set, 40 of 200 claims require review; in the validation set, 10 of 50 do. Three models have the following number of incorrect classifications:
| Model | Training errors | Validation errors |
|---|---|---|
| A | 5 of 200 | 9 of 50 |
| B | 22 of 200 | 7 of 50 |
| C | 8 of 200 | 15 of 50 |
Method. Convert each error count to a proportion before comparing the sets, then interpret the validation ranking alongside the train-validation gaps.
Step 1: Calculate training error rates. Model A has \(5/200=0.025\), or 2.5%. Model B has \(22/200=0.11\), or 11%. Model C has \(8/200=0.04\), or 4%.
Step 2: Calculate validation error rates. Model A has \(9/50=0.18\), or 18%. Model B has \(7/50=0.14\), or 14%. Model C has \(15/50=0.30\), or 30%. Model B has the lowest observed validation error rate.
Step 3: Compare the gaps. The validation-minus-training differences are 18%-2.5%=15.5 percentage points for A, 14%-11%=3 percentage points for B, and 30%-4%=26 percentage points for C. Model A's very low training error does not translate into the lowest validation error. Model C has high validation error and a large gap, evidence consistent with overfitting. Model B has the best validation error and smallest gap among the three.
The review class is 40/200=20% of the training data and 10/50=20% of the validation data, so the overall class proportions match. However, overall error rates do not show how many review cases each model misses. Before recommending a model, inspect class-specific results, such as the number of review cases correctly identified, if detecting those cases is important.
Common mistake and correction. Choosing Model A because it has only 5 training errors ignores that it makes 9 validation errors out of 50, compared with Model B's 7. Model B is the observed validation winner under overall classification error, but a validation set of 50 is limited evidence. A larger representative evaluation and class-specific performance would help assess whether that advantage matters for the intended use.
Small Validation Sets and Uneven Losses
Validation error is an average, so its reliability depends on more than the numerical size of the average. With few observations, a small number of difficult cases can dominate the result. Looking at case-level losses can show whether a ranking is broadly consistent or driven by one unusual observation. It does not, by itself, resolve uncertainty; additional representative observations may be needed.
Worked Example: Checking Whether a Small Validation Set Is Decisive
A hypothetical equipment-pricing team predicts repair cost in thousands of dollars. On the same four validation cases, the squared prediction errors from two models are as follows:
| Validation case | Model P squared error | Model Q squared error |
|---|---|---|
| 1 | 1 | 4 |
| 2 | 4 | 4 |
| 3 | 9 | 4 |
| 4 | 16 | 4 |
Method. Compute each validation MSE, compare the paired case-level losses, and state how the small sample limits the decision.
Step 1: Compute Model P's validation MSE. Its total squared error is \(1+4+9+16=30\). Divide by four cases: \(30/4=7.5\).
Step 2: Compute Model Q's validation MSE. Its total squared error is \(4+4+4+4=16\). Divide by four cases: \(16/4=4\). Model Q therefore has the lower observed validation MSE, 4 compared with 7.5.
Step 3: Inspect the paired differences. Subtract Q's squared error from P's for each case: \(1-4=-3\), \(4-4=0\), \(9-4=5\), and \(16-4=12\). The differences sum to \(-3+0+5+12=14\); their average is \(14/4=3.5\), equal to the MSE difference \(7.5-4=3.5\). P is better on one case, tied on one, and worse on two, with the largest difference on case 4.
The observed average favors Q, but it is based on only four cases, and one case contributes substantially to the difference. It would be too strong to conclude from this result alone that Q will consistently predict better in the target population. The case-level comparison identifies why the ranking occurred; a larger representative evaluation would provide stronger evidence.
Common mistake and correction. Treating the lower MSE as conclusive ignores the very small validation set and the uneven paired losses. Report the observed ranking accurately, then qualify how much it supports a population-level conclusion.
A Practical Interpretation Sequence
When train and validation results are presented together, use a fixed sequence rather than interpreting one number in isolation:
Confirm that all models use the same outcome, loss, validation observations, and evaluation procedure. Check that the validation population is relevant to the intended prediction setting.
Identify the observed validation winner under the specified loss. Do not substitute training error for the quantity used to assess held-out performance.
High errors on both sets can suggest underfitting; a much lower training error than validation error can suggest overfitting. State these as possible explanations, not certainties.
Consider validation-set size, case-level losses, class-specific performance where relevant, and whether the data reflect the target population. An untouched test set can provide a final evaluation after selection.
The distinction between an observed result and a broader conclusion is central. “Model B had the lowest validation error on this split” reports what was measured. “Model B will perform best on future cases” is a stronger claim that needs evidence that the split is informative and that the comparison is not an artifact of the particular observations.
Check Your Understanding
For each problem, compare the stated results and qualify what the evidence supports.
- Two regression models have training and validation MSEs of 16 and 25 for Model A, and 4 and 9 for Model B. Which has lower validation MSE, and what pattern in the results might suggest underfitting or overfitting? (Answer: Model B has lower validation MSE, 9 versus 25. A's higher errors on both sets may suggest underfitting relative to B; B's 5-unit gap alone does not establish overfitting.)
- A classifier makes 12 errors on 120 training cases and 6 on 30 validation cases. Calculate both error rates and state what their difference suggests. (Answer: Training error is 10%; validation error is 20%; the 10-percentage-point gap is evidence consistent with overfitting, but is not conclusive.)
- Model R has training MSE 16 and validation MSE 36; Model S has training MSE 36 and validation MSE 49. Calculate their validation RMSEs and identify the validation winner. (Answer: R has validation RMSE \(\sqrt{36}=6\); S has \(\sqrt{49}=7\). R has lower validation error.)
- On 40 validation cases, Classifier A makes 8 errors and Classifier B makes 5. Which has lower overall validation error, and what additional evidence could matter if the positive class is rare? (Answer: A has 20% error and B has 12.5%, so B is lower overall; compare class-specific results, such as positive cases correctly detected and missed.)
- Two models' validation MSEs are 8 and 7, based on only five observations. What can be stated, and what would strengthen the comparison? (Answer: The second model has the lower observed validation MSE, 7 versus 8, on this split; a larger representative evaluation or an untouched test set would strengthen the conclusion.)