Why Plot Performance Against Sample Size?
A model's predictive performance depends partly on how much data was available to fit it. A model trained on a small sample may be sensitive to which observations happened to be included; with more observations, its predictions may change. A learning curve makes this relationship visible by plotting performance against training-set size.
A learning curve is not a plot of performance against model complexity. Instead, it holds the learning procedure and the evaluation measure fixed while varying the number of training observations. It usually displays both training performance and validation performance. Reading the two curves together helps distinguish a large training-validation gap from performance that remains limited on both data sets.
How to Construct and Read a Learning Curve
For each selected training size \(n\), fit the specified learning procedure using \(n\) observations and calculate the chosen measure on the training data and on validation data. The horizontal coordinate is \(n\); the vertical coordinate is the measure. For loss-based curves, it is useful to define the gap as validation loss minus training loss:
A positive gap means that the validation loss is higher than the training loss. A small gap means the two measured losses are close, but it does not by itself establish that predictions are good: both losses might be large. Conversely, a large gap indicates a difference between the observed training and validation performance, not the exact cause of that difference.
The vertical levels matter as much as the gap. Low losses on both sets suggest good performance under the chosen measure and evaluation conditions. High losses on both sets suggest limited performance under those conditions, even if the gap is small. Whether a loss is low or high needs context, such as its scale, an appropriate benchmark, and the prediction goal.
The points at different sample sizes are generally estimates, not exact properties of a learning algorithm. A curve may be estimated by repeating the fitting process on different training samples of each size and summarizing the resulting performance. If only one training sample is used at each size, the plotted points can reflect the particular observations selected. Validation results also vary with the validation observations.
Worked Example: A Shrinking Gap as Sample Size Grows
A hypothetical insurer fits the same claim-cost prediction procedure at training sizes \(n=25\), \(100\), and \(400\). It measures mean squared error (MSE), in squared thousands of dollars, and obtains the following values. Treat the listed values as the measured results for this example.
| Training size \(n\) | Training MSE | Validation MSE |
|---|---|---|
| 25 | 4.0 | 12.0 |
| 100 | 7.0 | 9.0 |
| 400 | 8.0 | 8.5 |
Step 1: Calculate the gaps. At \(n=25\), \(G(25)=12.0-4.0=8.0\). At \(n=100\), \(G(100)=9.0-7.0=2.0\). At \(n=400\), \(G(400)=8.5-8.0=0.5\). Each gap is positive, so validation MSE exceeds training MSE at every listed size.
Step 2: Compare the validation levels. Validation MSE falls from \(12.0\) to \(9.0\) to \(8.5\) as the training size increases. For these observations, predictive performance on the validation set improves by the loss criterion, with most of the reduction occurring between \(n=25\) and \(n=100\).
Step 3: Interpret the pattern cautiously. The observed gap narrows, and the two losses are close at \(n=400\). This is consistent with less separation at the largest listed size. It does not prove that the gap will be zero at a still larger size, or that \(8.5\) is the best achievable loss.
Common mistake and correction. Subtracting training loss from validation loss in the reverse order gives \(4.0-12.0=-8.0\) at \(n=25\). That negative result might be misread as training loss exceeding validation loss. The defined gap is validation minus training, so the correct value is \(12.0-4.0=8.0\), indicating higher validation loss. The method is to check the subtraction order before interpreting the sign.
Why Validation Loss Targets Generalization
Generalization concerns prediction on new observations from the population of interest. The rationale for using validation loss is that, under suitable sampling conditions, validation observations are separate from the data used to fit the model.
Let \(D_n\) denote a training sample of size \(n\), and let \(\hat f_{D_n}\) be the fitted prediction rule. Suppose the validation observations \(Z_1,\ldots,Z_m\) are independent of \(D_n\), are independently sampled from the same target population, and are evaluated using a specified loss function \(L\). Conditional on the fitted rule, define its population prediction risk as \(R(\hat f_{D_n})=E[L(Z,\hat f_{D_n})\mid D_n]\), where \(Z\) is a new observation from that population. The average validation loss is
Justification. Conditional on \(D_n\), the fitted rule is fixed. Each validation observation has the same target-population distribution and is independent of the fitted rule, so each loss has conditional expectation \(R(\hat f_{D_n})\). By linearity of expectation,
Thus, under the stated assumptions, validation loss is an unbiased estimate of the fitted rule's population risk conditional on the training sample. It is still subject to sampling variability. If validation observations are not representative of the target population, or are not independent of model fitting, this justification does not apply as stated.
Training loss is calculated on the observations used to fit the rule, rather than on independent observations. The fitting procedure may adapt to those observations, so training loss is not generally an unbiased estimate of population risk. This is why a small training loss alone is not enough to conclude that future predictions will be accurate.
A learning curve may also average performance over repeated training samples. Such an average describes expected performance across training samples of size \(n\), if the repetitions are drawn according to the relevant sampling process. It is distinct from the conditional risk for one particular fitted model.
Worked Example: Calculating Validation MSE and Its Interpretation
A hypothetical pricing team fits a model on \(n=60\) policies. For four independent validation policies, actual annual costs, in thousands of dollars, are \(2,4,6,8\), and the model predictions are \(3,3,5,7\), respectively. Calculate validation MSE and interpret what this one validation result can establish.
Step 1: Find each prediction error. Actual cost minus predicted cost gives \(2-3=-1\), \(4-3=1\), \(6-5=1\), and \(8-7=1\), all in thousands of dollars.
Step 2: Square the errors. The squared errors are \((-1)^2=1\), \(1^2=1\), \(1^2=1\), and \(1^2=1\), in squared thousands of dollars.
Step 3: Average the squared errors. The sum is \(1+1+1+1=4\), and there are four validation observations. Therefore, validation MSE is \(4/4=1\) squared thousand dollars.
Interpretation. Under independent sampling from the target population, this average estimates the fitted model's population risk for squared-error loss. It does not equal the exact population risk, because only four validation policies were observed. Nor can it be compared with training MSE unless the training MSE is supplied and calculated using the same loss definition and units.
Common mistake avoided. The errors must be squared before averaging for MSE. Averaging the signed errors here gives \((-1+1+1+1)/4=0.5\), which is mean error, not MSE. The correct calculation averages \(1,1,1,1\), giving MSE \(1\).
Observed Curves Are Not Guaranteed to Be Smooth
A learning curve often has a recognizable broad pattern, but individual plotted points need not move monotonically. When the training sample grows, the added observations may be more difficult to predict, or the composition of the sample may change. As a result, observed training loss can rise, and validation loss can rise or fall. A rising training loss does not by itself mean that the model-fitting process has failed.
This differs from a guarantee about training error in particular settings. For example, when fitting nested model classes by minimizing the same loss on the same fixed training data, the minimum training loss cannot increase as the model class expands. A learning curve changes the training data as \(n\) changes; the fixed-data guarantee does not imply that its plotted training loss must decrease.
When several fits are made at each size, summarize the results clearly. A mean curve describes average observed performance across those fits; ranges or other variability summaries can show how much the results differ. Do not treat successive points as independent without justification: if the same validation observations are used at each size, their losses can be related.
Worked Example: Summarizing Repeated Fits
A hypothetical analyst repeats model fitting three times at each of \(n=40\) and \(n=160\), using MSE throughout. The training MSE values are \(5,6,7\) at \(n=40\) and \(6,7,8\) at \(n=160\). The corresponding validation MSE values are \(9,10,11\) and \(7.5,8,8.5\). Calculate the mean training and validation MSE at each size, then compare the mean gaps.
Step 1: Average at \(n=40\). The mean training MSE is \((5+6+7)/3=18/3=6\). The mean validation MSE is \((9+10+11)/3=30/3=10\). The gap between the means is \(10-6=4\).
Step 2: Average at \(n=160\). The mean training MSE is \((6+7+8)/3=21/3=7\). The mean validation MSE is \((7.5+8+8.5)/3=24/3=8\). The gap between the means is \(8-7=1\).
Step 3: State what the summaries show. Mean validation MSE is lower at the larger training size, decreasing from \(10\) to \(8\). The gap between mean validation and mean training loss also narrows, from \(4\) to \(1\). Mean training MSE rises from \(6\) to \(7\); this does not contradict the learning-curve interpretation because the training samples differ across sizes.
Common mistake avoided. It would be incorrect to state that the larger training size must have a lower training MSE because it contains more observations. That monotonicity is not guaranteed when the observations used to calculate the loss change. The method is to calculate each plotted summary from its own results, then interpret the pattern actually observed.
What a Learning Curve Can—and Cannot—Say
The level and separation of the curves answer different questions. For a loss measure, the validation level summarizes estimated predictive loss at that sample size; the gap describes how much higher or lower that loss is than training loss. Read both before drawing a conclusion. In particular:
- Low validation loss and a small gap: the observed predictive loss is low relative to the relevant scale or benchmark, and training and validation results are close.
- High validation loss and a small gap: the two results are similar, but predictions may still be inadequate. A small gap is not synonymous with good prediction.
- Low training loss and substantially higher validation loss: the fitted rule performs better on its training observations than on the validation observations. The gap signals a generalization concern, but does not identify its cause on its own.
- Changing validation loss as \(n\) increases: the observed results indicate how performance changed for the fitted procedure and samples considered. They do not guarantee the same pattern at larger sizes.
The curve also depends on the performance measure. For MSE or another loss, smaller is better; for accuracy or another score, larger is better. A gap must be defined with an explicit subtraction order, and conclusions should respect the measure's direction. Comparisons are meaningful only when the same measure and compatible evaluation conditions are used.
Check Your Understanding
For each problem, state calculations and interpret the result in terms of the specified measure.
- A loss-based curve reports training MSE \(3\) and validation MSE \(7\) at \(n=30\), then training MSE \(5\) and validation MSE \(6\) at \(n=120\). Find both gaps and describe how the observed validation loss and gap changed. (Answer: Gaps \(4\) and \(1\); validation MSE fell from \(7\) to \(6\), and the gap narrowed.)
- A classification procedure has training accuracy \(0.88\) and validation accuracy \(0.80\) at \(n=50\), then training accuracy \(0.84\) and validation accuracy \(0.83\) at \(n=200\). Using training accuracy minus validation accuracy as the gap, calculate both gaps and state the validation-accuracy change. (Answer: Gaps \(0.08\) and \(0.01\); validation accuracy rose by \(0.03\).)
- At \(n=50\), three repeated fits have training MSE \(4,5,6\) and validation MSE \(8,10,12\). At \(n=200\), the values are training MSE \(5,6,7\) and validation MSE \(7,8,9\). Find each mean training MSE, mean validation MSE, and gap between means. (Answer: At \(n=50\), means \(5\) and \(10\), gap \(5\); at \(n=200\), means \(6\) and \(8\), gap \(2\).)
- A learning curve has validation MSE \(22\) and training MSE \(21\) at one size, with nearly the same values at a larger size. What does the small gap support, and what can it not establish without a benchmark or further context? (Answer: The observed training and validation losses are close; the small gap alone cannot establish that predictive performance is good.)
- For a fixed fitting procedure, observed training MSE increases from \(4\) at \(n=40\) to \(5\) at \(n=160\), while validation MSE falls from \(9\) to \(7\). Is this necessarily a contradiction or a calculation error? (Answer: No; the training observations change with sample size, so observed training MSE need not decrease, while the stated validation loss improves.)