Why the Three Sets Have Different Jobs
A predictive model is developed using data, but the data used to fit it cannot by itself provide an independent check of how well it predicts new observations. Model development also involves choices: which candidate to keep, how complex it should be, or which settings to use. If the same observations both drive those choices and provide the final performance report, the reported score can reflect those choices as well as the model’s predictive ability.
A training set, a validation set, and a test set address different parts of this problem. The training set fits candidate models. The validation set guides choices among candidates. The test set is held aside to estimate the performance of a final model after those choices are fixed. This division is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
These are roles, not special kinds of observations. Ideally, the sets represent the same prediction population and are separated so that the observations used for one role do not also serve another. The prediction target and evaluation loss should be specified for the task. For example, a model used to predict dollar amounts might be evaluated by mean absolute error, while a classification task might use a classification loss. The appropriate measure depends on the prediction goal.
What Each Set Can Tell You
| Set | What is done with it | What its loss mainly describes |
|---|---|---|
| Training | Fit each candidate model | Performance on observations used in fitting |
| Validation | Compare candidates and guide development choices | Performance on observations used to guide those choices |
| Test | Evaluate the chosen model after development decisions are fixed | An estimate of performance on new observations from the target population |
Training loss is useful for seeing how well a fitted candidate matches its training data, but it is not an independent estimate of its future performance. Validation loss is useful for choosing between candidates, but the candidate is chosen partly because of its validation results. Test loss has a distinct purpose: it gives a final assessment, provided the model and the choices that define it were fixed without using the test results.
The same evaluation loss should be used when comparing candidates for a given prediction goal. Otherwise, a lower score may simply reflect a change in what is being measured. The earlier material on matching evaluation loss to the prediction goal addresses that choice; here the focus is which observations contribute to each stage.
Why the Test Average Estimates Future Loss
Let \(f\) be a model that has already been fixed, and let \(L(Y,f(X))\) be the loss incurred when the outcome is \(Y\) and the predictors are \(X\). The population prediction risk is the expected loss for a new observation from the target population. If the test observations are independent draws from that population and were not used to select or fit \(f\), their average loss estimates that risk.
The justification is the expectation of a sample average. With \(f\) fixed, each test loss has expected value \(R(f)\). Therefore, by linearity of expectation,
This result explains why independence from model development matters: it treats \(f\) as fixed when the test losses are averaged. The test average is still an estimate, not a guarantee that future loss will equal the reported value. A finite test set can give a noisy estimate, and the estimate applies to the population represented by the test observations.
A Development Sequence
Estimate the parameters of each candidate using the training set. Do not describe the resulting training loss as final evidence of future performance.
Compare candidates with the chosen loss and settle on a model and its development choices.
Calculate the chosen loss on the held-out test observations and report it as an estimate of performance, with the evaluation context stated.
The important feature is not a particular allocation of observations but the separation of purposes. A data set called “test” does not provide a final evaluation if its results are used to change the model and the changed model is then presented as though it had been evaluated independently. Conversely, a validation set is not a final test set merely because it was held out during fitting; it has already informed development decisions.
Worked Example: Comparing Claim-Cost Models
A hypothetical insurer develops models to predict claim costs for a new type of optional home-repair coverage. Lower squared prediction loss is preferred. Three candidates are fitted on the training set. Their training losses are: Model A, 4; Model B, 4; and Model C, 2, in squared-cost units per observation. Their validation losses on four separate observations are as follows:
| Candidate | Validation losses | Mean validation loss |
|---|---|---|
| A | 5, 6, 7, 6 | ? |
| B | 4, 5, 5, 6 | ? |
| C | 8, 7, 6, 7 | ? |
Method: compare mean validation loss, then evaluate the selected model once on the held-out test set. For Model A, the validation losses sum to \(5+6+7+6=24\), so its mean is \(24/4=6\). For Model B, they sum to \(4+5+5+6=20\), giving \(20/4=5\). For Model C, they sum to \(8+7+6+7=28\), giving \(28/4=7\).
Model B has the smallest validation loss, so it is selected. This choice is not based on the training losses: Model C has the lowest training loss, but its validation result is the worst of the three. The validation set has served its role by guiding the choice.
After the choice is fixed, the test losses for Model B are \(7,5,6,6,6\). Their sum is \(7+5+6+6+6=30\), and the test mean is \(30/5=6\) squared-cost units per observation. This is the final test estimate for the selected model in this example. It is not a reason to switch to another candidate, because the test set’s role is evaluation after the choice is fixed.
Check: Model B’s test losses sum to 30 across five observations, so their average is 6. The common mistake avoided is selecting Model C solely because its training loss is smallest.
Worked Example: Selecting a Repair-Time Predictor
A hypothetical service plan predicts the time, in dollars of labor cost, required for a covered appliance repair. A development team uses mean absolute error (MAE), measured in dollars per repair. Two candidate models are fitted on training data. Their training MAEs are $10 and $8 for Models A and B, respectively. On four validation repairs, the absolute errors are \(12,18,10,20\) for A and \(16,14,13,15\) for B. On five separate test repairs, the selected model’s absolute errors are \(11,17,16,12,14\).
Method: use validation MAE to choose the candidate, then compute the test MAE for that fixed choice. Model A’s validation errors sum to \(12+18+10+20=60\), so its validation MAE is \(60/4=\$15\). Model B’s errors sum to \(16+14+13+15=58\), so its validation MAE is \(58/4=\$14.50\). Since $14.50 is lower than $15, choose Model B.
Now calculate the test MAE for Model B. The five test errors sum to \(11+17+16+12+14=70\), and \(70/5=\$14\). Thus the final test estimate is $14 per repair. Model B’s training MAE of $8 is not the reported final estimate: those errors came from observations used to fit the model.
Check: The validation comparison gives \(15\) versus \(14.50\), and the selected model’s test average is \(14\). The common mistake avoided is reporting the lower training MAE as if it estimated performance on new repairs.
Worked Example: A Classification Model and Brier Loss
A hypothetical insurer predicts whether an optional equipment-protection claim will occur. For a binary outcome \(Y\), the Brier loss for a predicted probability \(p\) is \((Y-p)^2\), where \(Y=1\) means a claim occurs and \(Y=0\) means it does not. Two candidate probability models are fitted on training data. Their validation Brier losses are \(0.08\) for Model A and \(0.05\) for Model B, so the team fixes Model B as its choice. The outcomes and Model B’s predicted probabilities on four untouched test observations are:
| Test observation | Outcome \(Y\) | Predicted probability \(p\) |
|---|---|---|
| 1 | 1 | 0.8 |
| 2 | 0 | 0.3 |
| 3 | 1 | 0.6 |
| 4 | 0 | 0.1 |
Method: calculate the chosen model’s Brier loss for each test case and average. For observation 1, \((1-0.8)^2=0.2^2=0.04\). For observation 2, \((0-0.3)^2=(-0.3)^2=0.09\). For observation 3, \((1-0.6)^2=0.4^2=0.16\). For observation 4, \((0-0.1)^2=(-0.1)^2=0.01\).
The four losses sum to \(0.04+0.09+0.16+0.01=0.30\). The mean test Brier loss is \(0.30/4=0.075\). This is the test estimate for Model B under the stated loss. It is not a validation score: Model B was chosen using validation information, while these test outcomes were reserved for the final evaluation.
Check: The squared errors are \(0.04, 0.09, 0.16, 0.01\), whose mean is \(0.075\). The common mistake avoided is treating a validation result used to choose the model as though it were an independent final evaluation.
Common Interpretation Traps
- Calling training loss a generalization estimate. Training observations helped determine the fitted model, so their loss describes fit to those observations, not an independent assessment of new data.
- Using validation loss as the final report after choosing with it. Validation supports development choices. The chosen candidate may have benefited from favorable validation results, so those results do not have the same interpretation as a held-out test estimate.
- Changing the model after seeing test performance. Once test results guide a new choice, they have become part of development. The resulting score no longer evaluates that changed choice independently.
- Ignoring what population the split represents. An average test loss estimates performance for observations like those in the test set. It need not describe a different population or setting.
- Changing the loss across stages without explanation. A validation comparison under one loss and a final report under another answer different questions. State the metric and its units or interpretation.
Check Your Understanding
For each situation, identify the relevant set or calculate the requested result.
- Two fitted models have training losses of 3 and 2. Their validation losses are 5 and 4. Which set supports choosing between them, and which model has the lower validation loss? (Answer: The validation set supports the choice; Model 2 has the lower validation loss, 4.)
- A fixed model has test absolute errors of \(6, 10, 8, 12\) dollars. Calculate its test MAE. (Answer: \((6+10+8+12)/4=\$9\).)
- A team chooses one of three candidates using validation loss, then evaluates the selected candidate on a separate test set. Which score is intended as the final estimate of the selected candidate’s performance, and why? (Answer: The test-set score, because the candidate was fixed without using that set.)
- A binary-outcome model gives \(p=0.7\) when \(Y=1\), and \(p=0.2\) when \(Y=0\). Calculate the mean Brier loss across these two observations. (Answer: \(((1-0.7)^2+(0-0.2)^2)/2=(0.09+0.04)/2=0.065\).)
- A model’s test mean loss is 11. After reviewing it, the team changes the model and reports the same test score as the final evaluation of the changed model. What is the problem? (Answer: The test results influenced the changed model, so that score is no longer an independent final evaluation of the changed choice.)