Reading a Learning Curve as a Pattern
A learning curve shows how a model’s error changes as the number of training observations changes. Its value is not just in showing whether an error went up or down. Read the training error, validation error, and the gap between them together, and compare their levels with a meaningful benchmark. This helps distinguish a pattern consistent with persistent bias from one in which a variance-related gap may be shrinking.
A curve is evidence about a particular learning procedure, data setting, and error measure. It is not a guarantee that more data will improve performance, nor does one irregular point establish that a model has changed its underlying behavior. Conclusions should reflect the overall pattern across sample sizes.
For an informative comparison, keep the response definition, loss measure, candidate learning procedure, and validation sample fixed as \(n\) changes. Ideally, the training sets are nested: the smaller training set is contained in the larger one. If the validation sample or loss measure changes between points, apparent curve movement may reflect those changes rather than sample size.
A benchmark also matters. For example, a simple prediction rule evaluated with the same loss can help establish whether an error level is high in context. A validation error can be much higher than training error yet still be acceptable relative to a useful benchmark; it can also be close to training error while both are poor.
Three Features to Inspect
Use the following sequence rather than assigning a label from a single point.
Ask whether training and validation errors are low or high relative to a meaningful benchmark or practical target. Two similarly high errors can indicate that the procedure is not capturing important structure.
Use \(G(n)=E_{\text{val}}(n)-E_{\text{train}}(n)\). A substantial positive gap is consistent with training performance that does not carry over well to validation data.
See whether validation error falls, levels off, or moves irregularly as \(n\) grows. A falling validation error and narrowing gap suggest more observations may help; a flat curve suggests limited improvement over the observed range.
A pattern of high training and validation errors with a small gap is consistent with persistent bias: the fitted procedure performs poorly even on the data used to fit it, and validation performance is similarly poor. More observations alone may not solve the main problem if the procedure continues to miss important structure. The word “consistent” is important: a curve does not uniquely identify the cause.
A pattern of low training error, higher validation error, and a gap that shrinks with \(n\) is consistent with a variance-related problem that is easing as more data stabilize the fitted model. If validation error is still declining at the largest observed sample size, additional observations may improve it. This is a reasoned expectation, not a certainty.
Curves need not move smoothly. Training error often rises as the sample grows because a model has to accommodate a broader set of observations, but that direction is not a universal mathematical rule for average training error. Validation error can also rise at an intermediate sample size because the newly added training observations alter the fit. The overall trend and the size of the changes matter more than a single step.
Why More Data Can Reduce a Variance Component
A simple squared-error calculation illustrates why increasing sample size can lower prediction error while leaving a floor that more data cannot remove. Suppose a prediction target has a fixed component of squared bias \(b^2\), a variance component \(c/n\), and independent outcome noise with variance \(\sigma^2\). Under these simplifying assumptions, the expected squared prediction error is $$ R(n)=b^2+\frac{c}{n}+\sigma^2. $$ The expression is illustrative, not a general formula for every learning curve.
For the justification, write the prediction error as the sum of estimation error and outcome noise. If the prediction estimator has mean \(f(x)+b\), variance \(c/n\), and the outcome noise has mean zero and variance \(\sigma^2\), then the squared prediction error has expectation $$ \begin{aligned} E[(Y-\widehat f_n(x))^2] &=E[(f(x)+\varepsilon-\widehat f_n(x))^2]\\ &=(E[\widehat f_n(x)]-f(x))^2+\operatorname{Var}(\widehat f_n(x))+\sigma^2\\ &=b^2+\frac{c}{n}+\sigma^2. \end{aligned} $$ The cross terms vanish because the noise has mean zero and is independent of the estimator. As \(n\) increases, \(c/n\) decreases, but the fixed bias and noise terms remain. This explains why validation error may improve and then approach a nonzero floor. Real learning curves may depart from this simplified pattern because bias, variance, or the data distribution can behave differently.
Worked Example: High Errors With a Narrow Gap
A hypothetical claims-severity model is trained at four sample sizes. The error measure is mean squared prediction error, and a simple benchmark has validation error 12 at every sample size. The candidate model gives the following errors.
| Training size \(n\) | Training error | Validation error |
|---|---|---|
| 50 | 14 | 24 |
| 100 | 15 | 25 |
| 200 | 15.5 | 25.5 |
| 400 | 16 | 26 |
Method. Calculate the gap at each sample size, compare validation errors with the benchmark, and then describe the trend.
At \(n=50\), \(G(50)=24-14=10\). At \(n=100\), \(G(100)=25-15=10\). At \(n=200\), \(G(200)=25.5-15.5=10\). At \(n=400\), \(G(400)=26-16=10\). The gap stays at 10 across all four sizes.
Validation error rises from 24 to 26, a change of \(26-24=2\). It is also 12 points above the benchmark at \(n=50\), since \(24-12=12\), and 14 points above it at \(n=400\), since \(26-12=14\). Both training and validation errors are elevated relative to that benchmark, while the gap remains constant.
Conclusion. This pattern is consistent with persistent bias rather than a variance-related gap that is being reduced by more observations. The curve gives little evidence that simply increasing the sample from 50 to 400 improves validation performance.
Common mistake avoided. Calling the gap “small” just because it does not change would confuse a stable gap with a narrow gap. Here the gap is 10, and the validation error is far above the stated benchmark; its constancy does not make the model’s errors acceptable.
Worked Example: A Shrinking Gap and Falling Validation Error
A hypothetical model predicts the time until a policyholder completes a digital application. Its training and validation mean squared errors are measured using the same outcome scale and the same validation sample.
| Training size \(n\) | Training error | Validation error |
|---|---|---|
| 40 | 5 | 25 |
| 80 | 7 | 18 |
| 160 | 8 | 13 |
| 320 | 9 | 10 |
Method. Compute each validation-minus-training gap and the total change in validation error.
The gaps are \(25-5=20\) at \(n=40\), \(18-7=11\) at \(n=80\), \(13-8=5\) at \(n=160\), and \(10-9=1\) at \(n=320\). Thus the gap narrows by \(20-1=19\) from the first to the last sample size.
Validation error falls by \(25-10=15\) over the same range. The training error rises by \(9-5=4\), while validation error falls at each listed step. The validation improvement is \(25-18=7\), then \(18-13=5\), then \(13-10=3\).
Conclusion. The falling validation error and narrowing gap are consistent with a variance-related gap that is reduced as sample size grows. Since validation error is still falling at \(n=320\), more observations may help, although the smaller stepwise improvements caution against assuming the same rate of progress will continue.
Common mistake avoided. It would be wrong to conclude that model performance worsens just because training error increases from 5 to 9. The prediction target is generalization, for which the validation error falls substantially in this example.
Worked Example: A Nonmonotone Validation Curve
A hypothetical insurer uses a model to predict the number of service calls following a policy change. The loss is mean squared error, and the same validation sample is used at every training size.
| Training size \(n\) | Training error | Validation error |
|---|---|---|
| 60 | 8 | 16 |
| 120 | 10 | 14 |
| 240 | 11 | 15 |
| 480 | 11.5 | 13.5 |
Method. Calculate the gaps, then distinguish the overall trend from the intermediate reversal.
The gaps are \(16-8=8\), \(14-10=4\), \(15-11=4\), and \(13.5-11.5=2\). From \(n=60\) to \(n=480\), validation error falls by \(16-13.5=2.5\), and the gap falls by \(8-2=6\). However, when the sample grows from 120 to 240, validation error rises by \(15-14=1\).
Conclusion. The overall pattern is consistent with some improvement as sample size grows and a narrowing gap, but it is not monotone. The intermediate increase is a reason to avoid claiming that every increase in sample size improves validation performance. The final validation error is lower than the initial value by 2.5, but these four points alone do not establish how much of the pattern is stable rather than due to sampling variation.
Common mistake avoided. Ignoring the intermediate increase would overstate the smoothness of the evidence. Treating that one increase as proof that more data are harmful would overreact in the opposite direction.
Common Traps in Diagnosis
- Reversing the gap. If training error is 7 and validation error is 18, the defined gap is \(18-7=11\), not \(7-18=-11\). A positive gap indicates higher validation error.
- Using the gap without the error levels. Similar errors do not mean good performance if both errors are high relative to a relevant benchmark.
- Calling every positive gap persistent bias. A sizable gap points to a difference between training and validation performance; it is not, by itself, a diagnosis of bias. The levels and their changes with sample size matter.
- Expecting every point to move in one direction. A validation curve can fluctuate. Describe the broad pattern and note reversals rather than treating a single step as decisive.
- Promising that more data will fix the problem. Additional observations may reduce a variance component, but they do not necessarily correct persistent bias or remove irreducible noise.
This interpretation is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. In an exam-style diagnosis, support the conclusion with computed gaps, changes in validation error, and an explicit qualification when the pattern is not decisive.
Check Your Understanding
For each problem, use the stated error measure consistently and base the diagnosis on both the error levels and their pattern across sample sizes.
- A hypothetical customer-retention model has training errors 6, 7, and 8 and validation errors 15, 12, and 10 at sample sizes 25, 50, and 100. Calculate the three gaps and describe the main pattern. (Answer: Gaps are 9, 5, and 2; validation error falls by 5 overall and the gap narrows, consistent with improvement as sample size grows.)
- A model has training errors 11, 13, and 14 and validation errors 22, 25, and 26 at sample sizes 40, 80, and 160. A benchmark has validation error 10. Calculate the gaps and state the cautious diagnosis. (Answer: Gaps are 11, 12, and 12; both errors are high relative to the benchmark and validation error rises, consistent with persistent bias rather than a shrinking variance-related gap.)
- At sample sizes 30, 60, and 120, a model’s training errors are 3, 5, and 6, while validation errors are 14, 10, and 8. Calculate the gaps and total validation-error reduction, then interpret. (Answer: Gaps are 11, 5, and 2; validation error falls by 6, consistent with a shrinking gap and possible benefit from more observations.)
- For a model at sample sizes 100, 200, and 400, training errors are 7, 8, and 9 and validation errors are 18, 13, and 14. Calculate the gaps and explain why the last step needs qualification. (Answer: Gaps are 11, 5, and 5; validation error falls by 5 and then rises by 1, so the overall improvement is not monotone and the latest increase alone does not establish a harmful effect from more data.)
- A candidate model’s validation errors at sample sizes 50, 100, and 200 are 20, 16, and 15. A benchmark’s validation error is 14. What are the reductions between successive sample sizes, and what can be concluded about the largest sample? (Answer: Reductions are 4 and 1; validation performance improves but the latest value remains 1 above the benchmark, and the smaller final reduction suggests possible flattening rather than guaranteed continued improvement.)