Reading the Training-Validation Gap

Why Read the Two Errors Together?

A model’s training error describes how well it fits the observations used to estimate it. Its validation error describes how well that fitted model predicts a separate set of observations not used to fit it. Neither number, by itself, gives a complete diagnosis. A low training error can mean that the model has captured useful structure, or that it has adapted too closely to the training sample. A high validation error can signal poor fit, but it can also arise because that particular validation sample is difficult or unrepresentative.

The comparison is most informative when both errors use the same loss measure, the validation cases represent the prediction population, and the fitting process did not use the validation outcomes. A training-validation gap is a diagnostic clue, not a universal cutoff or a proof about the model. This way of reading errors is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: Let $L(y,\hat{y})$ be a loss measuring the error from predicting $\hat{y}$ when the outcome is $y$. For $n_{\mathrm{tr}}$ training observations and $n_{\mathrm{val}}$ validation observations, define the mean training error and mean validation error as $$ E_{\mathrm{tr}}=\frac{1}{n_{\mathrm{tr}}}\sum_{i=1}^{n_{\mathrm{tr}}}L(y_i,\hat{y}_i), \qquad E_{\mathrm{val}}=\frac{1}{n_{\mathrm{val}}}\sum_{j=1}^{n_{\mathrm{val}}}L(y_j^{\mathrm{val}},\hat{y}_j^{\mathrm{val}}). $$ The signed training-validation gap is $G=E_{\mathrm{val}}-E_{\mathrm{tr}}$. A positive gap means the observed validation error exceeds the observed training error. The sign and size of $G$ must be interpreted in context.

For squared-error regression, one possible loss is $L(y,\hat{y})=(y-\hat{y})^2$, and the corresponding mean errors are mean squared errors. Other losses may be appropriate for other prediction goals. The arithmetic difference $G$ is meaningful only when the two errors are on the same scale and computed comparably. In particular, a squared-error value cannot be subtracted from an absolute-error value to diagnose a gap.

Reading Error Levels and the Gap

There is no rule that a particular numerical gap always indicates overfitting. Interpret both the error levels and their relationship, using a relevant benchmark or comparison when available.

Observed pattern	Reasonable interpretation	What it does not establish
Training and validation errors are both high and fairly similar	Possible underfitting, especially if both exceed a meaningful benchmark	Similarity alone does not show that the model is adequate
Training error falls and validation error also falls after adding flexibility	The added flexibility appears useful for prediction on this validation set	It does not guarantee that further flexibility will help
Training error falls while validation error rises	A warning pattern consistent with overfitting	One split does not prove that the more flexible model has worse population performance
Validation error is below training error	Possible sampling variation or differences in case difficulty	It does not violate a general rule; validation error need not exceed training error on every split

For nested model classes fit by minimizing the same training loss, adding available model choices cannot increase the minimum training error: the smaller class’s best-fitting model remains an available choice in the larger class. This explains why training error can fall as flexibility increases. Validation observations are not used for that minimization, however, so their error can fall, remain similar, or rise. A falling training error alone therefore cannot establish improved prediction.

Why Validation Error Estimates Prediction Performance

The justification for using validation error depends on a separation between fitting and evaluation. Suppose a fitted model $\hat{f}$ is determined using the training data, and the validation observations are independent draws from the same population as future prediction cases. For a fixed fitted model, define its population prediction risk as $R(\hat{f})=E[L(Y,\hat{f}(X))\mid\text{training data}]$. Each validation loss has this conditional expectation, so averaging the validation losses also has that expectation:

E[E_{\mathrm{val}}\mid\text{training data}] = \frac{1}{n_{\mathrm{val}}}\sum_{j=1}^{n_{\mathrm{val}}} E[L(Y_j^{\mathrm{val}},\hat{f}(X_j^{\mathrm{val}}))\mid\text{training data}] =R(\hat{f}).

Thus, under these conditions, validation error estimates the prediction risk of the particular fitted model. The equality is an expectation, not a claim that one observed validation error equals the population risk. A finite validation set produces a variable estimate. If the validation cases differ in difficulty by chance, its average loss can be higher or lower than the population risk.

The argument also fails if validation outcomes influenced fitting, preprocessing, or model choices: then the evaluation data are no longer independent of the procedure being evaluated. Even without such information use, a single split gives only one sample-based estimate. A gap should therefore be read as evidence with uncertainty, rather than as a deterministic verdict.

Worked Example: Similar Errors, but Both Above a Benchmark

A hypothetical pricing team fits a simple model to four training cases and evaluates it on five separate validation cases. It uses mean squared error throughout. The training squared losses are $4, 5, 3, 4$; the validation squared losses are $5, 3, 4, 6, 2$. For this exercise, a previously specified practical benchmark is a mean squared error of $2$. The loss values are hypothetical observations; no probability distribution is assumed.

Method. Compute both mean errors, then the signed gap, and compare each error with the stated benchmark.

The total training loss is $4+5+3+4=16$, so $E_{\mathrm{tr}}=16/4=4$.

The total validation loss is $5+3+4+6+2=20$, so $E_{\mathrm{val}}=20/5=4$.

The gap is $G=4-4=0$. Both errors exceed the benchmark of $2$, while the observed gap is zero. The pattern is consistent with possible underfitting: the model’s errors are high relative to the stated target on both sets, without a training-validation separation in this split.

Check. The validation total is $20$, and $20/5=4$, matching the training mean. The method avoids treating a small gap as proof that a model is adequate; the benchmark matters.

When Added Flexibility Appears Useful

A more flexible model can reduce error on the training cases and also improve validation predictions. In that case, the fact that its training error is lower is not, by itself, evidence against it. The validation comparison is the relevant indication that the added flexibility helped on observations not used to fit the model. Still, a single validation result is not a guarantee of future performance.

Worked Example: Lower Validation Error Despite a Wider Gap

A hypothetical analyst compares two regression models on the same four training cases and the same four validation cases, using mean squared error. The less flexible model has training squared losses $4, 1, 9, 4$ and validation squared losses $9, 1, 4, 4$. The more flexible model has training squared losses $1, 0, 4, 1$ and validation squared losses $4, 1, 1, 4$. These are hypothetical loss values; no probability model is assumed.

Method. Find each mean error, calculate each gap, and compare the validation means.

For the less flexible model, training loss totals $4+1+9+4=18$, giving $E_{\mathrm{tr}}=18/4=4.5$. Validation loss totals $9+1+4+4=18$, giving $E_{\mathrm{val}}=18/4=4.5$. Its gap is $4.5-4.5=0$.

For the more flexible model, training loss totals $1+0+4+1=6$, giving $E_{\mathrm{tr}}=6/4=1.5$. Validation loss totals $4+1+1+4=10$, giving $E_{\mathrm{val}}=10/4=2.5$. Its gap is $2.5-1.5=1.0$.

The flexible model lowers validation error from $4.5$ to $2.5$, an improvement of $4.5-2.5=2.0$. Its gap has widened from $0$ to $1.0$, but the validation error has improved. On this split, the added flexibility appears useful; the wider gap alone is not a reason to reject it.

Check. The flexible model’s validation losses sum to $10$, so its mean is $10/4=2.5$, below $4.5$. The method avoids equating any positive gap with overfitting.

A Widening Gap Is a Warning, Not a Verdict

A common overfitting pattern is that training error falls as a model becomes more flexible while validation error rises. The divergence is a warning that the extra flexibility may be fitting idiosyncrasies in the training data rather than structure that predicts well elsewhere. But “may” matters: validation outcomes themselves vary, and a single split can make one model look unusually good or bad.

Worked Example: A Noisy Validation Result at High Flexibility

A hypothetical analyst compares three nested regression models with increasing flexibility. All use mean squared error on the same training cases and the same five validation cases. Their training mean squared errors are $3.0$, $1.8$, and $0.4$, respectively. The first model has validation squared losses $3,3,3,3,4$; the second has $0,1,2,2,5$; the third has $0,1,1,6,7$. Treat the five validation losses for each fixed model as independent observations from the same prediction population with finite variance. These are hypothetical losses, not data from an actual insurer.

Method. Calculate the validation mean and gap for each model. For the third model, also calculate the validation sample standard error to illustrate the variation in a single-split estimate.

The first model’s validation total is $3+3+3+3+4=16$, so its validation mean is $16/5=3.2$. Its gap is $3.2-3.0=0.2$.

The second model’s validation total is $0+1+2+2+5=10$, so its validation mean is $10/5=2.0$. Its gap is $2.0-1.8=0.2$.

The third model’s validation total is $0+1+1+6+7=15$, so its validation mean is $15/5=3.0$. Its gap is $3.0-0.4=2.6$. Training error keeps falling, but validation error rises from $2.0$ to $3.0$ between the second and third models. This is a clear warning pattern consistent with overfitting.

For the third model, the validation mean is $3.0$. The squared deviations from this mean are $9,4,4,9,16$, which sum to $42$. The sample variance is $42/(5-1)=10.5$, so the sample standard deviation is $\sqrt{10.5}\approx3.2404$. The estimated standard error of the validation mean is $\sqrt{10.5/5}=\sqrt{2.1}\approx1.4491$.

The estimated standard error is substantial relative to the one-unit difference between the second and third validation means. This does not erase the warning, but it shows why one split cannot settle the comparison with certainty. The second model is preferable on the observed validation means; the third model’s apparently worse performance should be treated as evidence, not proof.

Check. The third model’s mean is $15/5=3.0$, and its sample variance uses $n-1=4$, not $n=5$, in the denominator. The method avoids turning a single observed increase in validation error into a certain claim about population performance.

Common Exam Traps

Calling every positive gap overfitting. A positive gap can occur even when added flexibility improves validation performance, as in the second worked example. Check the direction of validation error across candidates.
Ignoring absolute error levels. A small gap does not mean good fit. If both errors are large against a meaningful benchmark, underfitting may still be a concern.
Comparing unlike quantities. Subtracting a training mean absolute error from a validation mean squared error does not yield a meaningful training-validation gap.
Assuming validation error must exceed training error. Sampling variation and case difficulty can produce a lower validation error. The inequality is not guaranteed for a particular split.
Treating a single split as conclusive. The validation mean is an estimate with sampling variability. A gap is most useful as one diagnostic signal, interpreted with the design and other evidence in view.

Key takeaway: Read training and validation errors jointly. High, similar errors can suggest underfitting; falling errors on both sets can indicate useful flexibility; falling training error paired with rising validation error warns of possible overfitting. None of these patterns, especially from one split, is a verdict without context.

Check Your Understanding

For each problem, use the stated loss consistently and distinguish the observed pattern from a certain conclusion about population performance.

A hypothetical model has training squared losses $2,3,3,4$ and validation squared losses $1,2,4,3,5$. Calculate both mean squared errors and the signed gap. If a practical benchmark is $2$, what pattern is suggested? (Answer: $E_{\mathrm{tr}}=3$, $E_{\mathrm{val}}=3$, $G=0$; both exceed the benchmark, so possible underfitting.)
Model A has training mean squared error $2.4$ and validation mean squared error $3.1$. Model B has training mean squared error $1.5$ and validation mean squared error $2.2$. Calculate both gaps and state whether B’s added flexibility appears useful on this validation set. (Answer: Both gaps are $0.7$; B lowers validation error by $0.9$, so its added flexibility appears useful on this split.)
A flexible model has training squared losses $0,1,1,2$, and validation squared losses $2,2,3,3,5$. Calculate the two mean errors and gap. What is the cautious diagnosis? (Answer: $E_{\mathrm{tr}}=1$, $E_{\mathrm{val}}=3$, $G=2$; the pattern warns of possible overfitting but does not prove it.)
Four validation squared losses are $1,2,2,3$, while the training mean squared error is $1.5$. Calculate the validation mean, the gap, and the estimated standard error of the validation mean using the sample standard deviation. (Answer: $E_{\mathrm{val}}=2$, $G=0.5$, sample variance $=2/3$, and estimated standard error $=\sqrt{1/6}\approx0.4082$.)
A report gives training mean absolute error $0.8$ and validation mean squared error $2.5$. A candidate subtracts to report a gap of $1.7$ and diagnoses overfitting. Identify the error in this reasoning. (Answer: The metrics are on different scales, so $2.5-0.8$ is not a meaningful gap; recompute both errors using the same loss before interpreting them.)

What You'll Learn