Why Compare Performance Across Data Partitions?
A model can fit the data used to estimate it very well and still predict new observations poorly. A low training loss is therefore not enough to establish predictive value. To assess whether a model’s performance carries beyond its training data, compare its training loss with its validation and test losses, calculated using the same outcome, metric, and data partitions as competing models.
This comparison is useful across model classes. A GLM, an elastic net regression, and an ensemble may have different flexibility, but their performance differences are interpretable only when they are evaluated on comparable data and with the same loss definition. This tutorial develops a consistent way to read those patterns, aligned with the SOA Exam PA (Predictive Analytics) syllabus.
With this sign convention, a positive gap means the model has a higher loss away from its training observations. A small gap is not automatically good: a model can have similarly poor training and validation losses. Conversely, a large gap is a warning, but its interpretation depends on the absolute losses, the comparison model, and whether the partitions and metric are appropriate.
The loss must measure the same target in every comparison. For a regression task, mean squared error (MSE) is one possible loss. For binary classification, log loss is another. Do not compare a training MSE with a validation MAE, or a training loss from one outcome definition with a test loss from another. The gap would not describe a change in performance on a common scale.
How to Read a Generalization Gap
The most useful interpretation combines three questions: How low is the training loss? How much does loss rise on validation and test data? And how do those losses compare with an appropriate baseline or other candidate models? The pattern, rather than any one number, suggests whether the model may be too flexible, too restrictive, or sensitive to which observations are held out.
| Observed pattern | Possible interpretation | Important qualification |
|---|---|---|
| Low training loss; much higher validation and test loss | Possible overfitting: the model has fitted patterns that do not carry to held-out data. | A large gap is a warning, not proof of a particular cause. |
| Training and held-out losses are similar, but all are high relative to a useful benchmark | Possible underfitting: the model may be too restrictive to capture predictive structure. | Check that the benchmark and losses are comparable before concluding. |
| Validation loss is favorable but test loss is substantially worse, or the reverse | Possible instability across held-out partitions. | A single test result does not establish why the results differ. |
| Training and held-out losses are both low, with a modest gap | Encouraging evidence of predictive performance. | It does not guarantee similar performance in every future population. |
A GLM can have a gap if its specification or fitting process adapts to the training data. Regularization can constrain fitted coefficients and may reduce that gap, although stronger regularization can also leave the model too restrictive. A tree-based ensemble can fit complex patterns, sometimes producing very low training loss; that advantage is useful only if held-out loss also improves. These are possible patterns, not rules that assign a model class a fixed level of generalization.
Why Training Loss Can Be Optimistic
The training data help determine the fitted model, so training loss is not an independent check of predictive performance. One simple justification uses nested model classes. Suppose a flexible model class contains every model in a simpler class, and each is fit by minimizing the same training loss. Because the flexible class includes all the simpler class’s choices, its minimum training loss cannot be higher.
This inequality follows because minimizing over a set that contains all the choices in another set cannot produce a worse minimum. It concerns training loss only; it does not say that the flexible model has lower validation or test loss. The flexible model may use its additional options to fit noise in the training observations.
A related selection effect occurs when several candidate models are compared using the same validation data. Even if each candidate’s validation loss is a reasonable estimate when considered alone, selecting the candidate with the smallest observed validation loss favors a result that may partly reflect chance. Earlier material on optimism from repeated validation-based selection addresses that issue. In reading a gap, remember that validation loss may have influenced the choice of model, whereas the test partition should remain reserved for a final assessment.
A Practical Comparison Sequence
Check that candidate models use the same outcome, loss definition, train-validation-test partitions, and evaluation cases. Confirm that preprocessing was performed without using held-out information.
For each candidate, subtract training loss from validation loss and from test loss. Keep the direction consistent: positive values indicate greater loss on held-out data.
A small gap with high losses can indicate limited predictive value. A large gap combined with excellent training loss and poor held-out losses is consistent with overfitting.
Do not repeatedly change the model in response to test performance. If validation and test results disagree, report the disagreement rather than treating one number as a definitive explanation.
Worked Comparisons
Worked Example: Comparing Four Regression Models
A hypothetical insurer predicts claim cost using a GLM, an elastic net model, a random forest, and a boosted ensemble. All four are evaluated on the same partitions using MSE in thousands of squared dollars. The reported losses are:
| Model | Training MSE | Validation MSE | Test MSE |
|---|---|---|---|
| GLM | 1.02 | 1.10 | 1.12 |
| Elastic net | 0.96 | 1.01 | 1.04 |
| Random forest | 0.58 | 1.15 | 1.19 |
| Boosted ensemble | 0.74 | 0.99 | 1.02 |
Method. Calculate the validation and test gaps by subtracting training MSE from each corresponding loss.
For the GLM, \(G_{\text{val}}=1.10-1.02=0.08\) and \(G_{\text{test}}=1.12-1.02=0.10\). For elastic net, \(G_{\text{val}}=1.01-0.96=0.05\) and \(G_{\text{test}}=1.04-0.96=0.08\). For the random forest, the gaps are \(1.15-0.58=0.57\) and \(1.19-0.58=0.61\). For boosting, they are \(0.99-0.74=0.25\) and \(1.02-0.74=0.28\).
The random forest has the lowest training loss, but its validation and test losses are the highest in this comparison. Its much larger gaps are consistent with overfitting. The boosted ensemble has the lowest validation and test losses, while elastic net has smaller gaps. The GLM’s losses are higher than those of boosting on validation and test data, but its gap alone does not establish that the GLM is underfit.
The validation ranking favors boosting over elastic net: \(0.99<1.01\). The test ranking is also in favor of boosting: \(1.02<1.04\). These values describe this comparison, not a license to use the test partition repeatedly to tune the models. The common mistake avoided here is selecting the random forest because its training MSE is smallest.
Worked Example: Identifying Possible Underfitting Against a Baseline
A hypothetical maintenance team predicts repair costs. A baseline that predicts the training-set mean has training, validation, and test MSE values of 1.25, 1.22, and 1.24. A simple GLM has losses of 1.20, 1.18, and 1.21. A more flexible ensemble has losses of 0.71, 0.89, and 0.92. All values use the same partitions and outcome.
Method. First calculate each model’s gaps; then compare held-out losses with the baseline. For the GLM, \(G_{\text{val}}=1.18-1.20=-0.02\) and \(G_{\text{test}}=1.21-1.20=0.01\). The negative validation gap means validation MSE is slightly lower than training MSE; gaps need not always be positive. For the ensemble, \(G_{\text{val}}=0.89-0.71=0.18\) and \(G_{\text{test}}=0.92-0.71=0.21\).
On validation data, the GLM improves on the baseline by \(1.22-1.18=0.04\), or \(0.04/1.22\approx0.0328\), approximately 3.3%. The ensemble improves by \(1.22-0.89=0.33\), or \(0.33/1.22\approx0.2705\), approximately 27.0%. On test data, the corresponding reductions are \(1.24-1.21=0.03\) for the GLM and \(1.24-0.92=0.32\) for the ensemble.
The GLM has losses close to the baseline on both held-out partitions. This is evidence that it adds little predictive value in this comparison; if the ensemble’s gains are credible and relevant, the GLM may be too restrictive for the available predictive structure. The GLM’s small gap does not make it a successful predictor by itself. The common mistake avoided is calling a model well-generalized solely because its training and validation losses are close.
Worked Example: Reading Validation-Test Disagreement
A hypothetical clinic compares models predicting whether a patient will return for follow-up. The common metric is log loss, where lower values are better. The GLM has training, validation, and test losses of 0.52, 0.55, and 0.56. Elastic net has 0.50, 0.53, and 0.54. A random forest has 0.31, 0.66, and 0.57. Boosting has 0.43, 0.54, and 0.68.
Method. Calculate both gaps for the models whose validation and test results differ most. For the random forest, \(G_{\text{val}}=0.66-0.31=0.35\) and \(G_{\text{test}}=0.57-0.31=0.26\). Its held-out losses are both higher than its training loss, although its test result is better than its validation result by \(0.66-0.57=0.09\). For boosting, \(G_{\text{val}}=0.54-0.43=0.11\) and \(G_{\text{test}}=0.68-0.43=0.25\). Its test loss is worse than its validation loss by \(0.68-0.54=0.14\).
Elastic net has the lowest validation loss, \(0.53\), and the lowest test loss, \(0.54\), among the four candidates. Boosting’s validation result is close to elastic net, but its test loss is substantially higher. This is a warning that the apparent performance does not carry consistently across the two held-out partitions. With only these results, it is not justified to claim a specific cause or that boosting will always perform poorly. The common mistake avoided is treating one favorable validation score as proof of reliable generalization.
Common Traps
- Ranking models by training loss. Training loss reflects the data used to fit the model. Use held-out performance to assess generalization.
- Calling a small gap “good performance.” Similar training and validation losses can both be high. Compare absolute losses with an appropriate baseline.
- Assuming a large gap proves overfitting. It is consistent with overfitting, but the gap alone does not establish the cause. First check the metric, partitions, and preprocessing.
- Changing the test set into a tuning set. Repeatedly choosing models based on test results compromises its role as a final assessment.
- Ignoring the direction of a metric. These gap definitions assume a loss where smaller is better. For a metric where larger is better, define and interpret the comparison consistently rather than applying the loss-gap wording mechanically.
Check Your Understanding
For each question, use the stated loss convention: smaller values are better. Treat the partitions and metrics as comparable unless the question says otherwise.
- A GLM has training MSE 0.84, validation MSE 0.91, and test MSE 0.93. Calculate both gaps. (Answer: \(G_{\text{val}}=0.07\); \(G_{\text{test}}=0.09\).)
- An elastic net has training, validation, and test MSE values of 0.72, 0.88, and 0.90. A baseline has validation and test MSE values of 0.95 and 0.96. State whether the elastic net improves on the baseline on each held-out partition and calculate the training-to-test gap. (Answer: It improves by \(0.95-0.88=0.07\) on validation and \(0.96-0.90=0.06\) on test; \(G_{\text{test}}=0.90-0.72=0.18\).)
- A tree ensemble has training loss 0.40 and validation loss 0.76. A GLM has training loss 0.69 and validation loss 0.72. Which model has the larger validation gap, and what does that suggest? (Answer: The ensemble’s gap is \(0.76-0.40=0.36\), versus \(0.72-0.69=0.03\) for the GLM; the ensemble’s pattern is more consistent with overfitting, but the gap alone is not proof.)
- A model has validation log loss 0.48 and test log loss 0.63. Give one appropriate interpretation and one conclusion that is not justified by these values alone. (Answer: The held-out results disagree substantially, so performance appears partition-sensitive; the values alone do not establish the cause or prove the model will fail in every future population.)
- A model’s training and validation MSE are both 1.30, while a relevant baseline’s validation MSE is 0.92. Is the small gap sufficient to call the model successful? (Answer: No. The gap is \(1.30-1.30=0\), but its validation loss is worse than the baseline’s by \(1.30-0.92=0.38\), so the small gap does not establish useful predictive performance.)