Why Reserve a Test Set?
A modeling procedure may appear accurate because it has adapted to the data used to fit or choose it. A test set has a different role: it provides a final evaluation on observations that were not used to fit the model or make modeling decisions. This distinction matters when assessing whether a predictive procedure may generalize to future cases.
Training data are used to fit model parameters. Validation data can help choose among candidate procedures or settings. Once those choices are settled, a test set can be used once for a final performance assessment. The test result is evidence about the procedure as finalized; it is not another opportunity to choose a model, alter a threshold, or adjust a tuning parameter. This use of a held-out test set is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
What the Test Result Estimates
Let the finalized fitted prediction rule be \(\hat{f}\), and let \(L(Y,\hat{f}(X))\) be the loss for an observation with predictors \(X\) and outcome \(Y\). For squared-error regression, one possible loss is \(L(Y,\hat{f}(X))=(Y-\hat{f}(X))^2\). The procedure’s generalization risk is its expected loss on a new observation from the population of interest.
If the test observations are representative of the intended prediction population and are independent of the data and decisions that produced \(\hat{f}\), the average test loss estimates this risk:
where \(m\) is the number of test observations. This is an estimate, not a guarantee that future loss will equal the reported value. Its accuracy depends in part on test-set size and the variability of individual losses. A small test set can produce a noisy estimate.
Why the Test Set Must Not Guide Tuning
The reason for reserving the test set can be seen from the expectation of its average loss. Suppose \(\hat{f}\) is fixed without using test observations, and test observations are independent draws from the target population. Conditional on that fixed \(\hat{f}\), each test loss has expected value \(R(\hat{f})\). By linearity of expectation:
For test observation \(i\), write \(L_i=L(Y_i,\hat{f}(X_i))\). Under the stated conditions, \(E[L_i\mid\hat{f}]=R(\hat{f})\).
The test-set estimate is \(m^{-1}\sum_{i=1}^{m}L_i\).
\(E[m^{-1}\sum_i L_i\mid\hat{f}]=m^{-1}\sum_iE[L_i\mid\hat{f}]=m^{-1}\sum_iR(\hat{f})=R(\hat{f})\). Thus the average test loss is unbiased for the risk of this fixed procedure under these conditions.
That justification no longer applies in the same way if the analyst uses test performance to choose among models and then reports the chosen model’s test score as a final, independent assessment. The chosen model is now connected to the test results: the test data helped determine which procedure is being evaluated. In particular, when several models have similar true performance, choosing the one with the smallest observed test loss tends to favor a model whose test estimate benefited from favorable random variation.
Using a test score to decide what to change, then reporting the same test score as though it were untouched, confuses development with evaluation. If the model, tuning, threshold, or other decision is changed after inspecting the test result, the test set has become part of the development process. A new independent evaluation source is needed for a fresh final assessment.
Worked Examples
Worked Example: Final Squared-Error Assessment
A hypothetical insurer develops a model to predict the paid amount of a claim, measured in thousands of dollars. A validation set was used to select the procedure, and all choices have now been fixed. On five previously reserved test claims, the observed and predicted amounts are as follows. Calculate test MSE, test RMSE, and test MAE.
| Claim | Observed | Predicted | Residual (observed − predicted) | Squared residual | Absolute residual |
|---|---|---|---|---|---|
| 1 | 10 | 11 | −1 | 1 | 1 |
| 2 | 12 | 10 | 2 | 4 | 2 |
| 3 | 15 | 17 | −2 | 4 | 2 |
| 4 | 20 | 18 | 2 | 4 | 2 |
| 5 | 23 | 25 | −2 | 4 | 2 |
Step 1: Calculate the test MSE. The squared residuals sum to \(1+4+4+4+4=17\). There are five test claims, so the mean squared error is \(17/5=3.4\) thousand-dollars squared.
Step 2: Calculate the test RMSE. Taking the square root gives \(\sqrt{3.4}=1.8439\) thousand dollars, rounded to four decimal places. The square root returns the measure to the outcome’s original units.
Step 3: Calculate the test MAE. The absolute residuals sum to \(1+2+2+2+2=9\), so the mean absolute error is \(9/5=1.8\) thousand dollars.
Interpretation and check. On this test set, the average squared error is 3.4 thousand-dollars squared, and the root mean squared error is about $1,843.90. The MAE is $1,800. These summaries describe the finalized procedure’s performance on the held-out observations; five claims are too few to imply that future errors will be exactly these values. The common mistake is to report MSE as though it were in thousands of dollars. It is in squared units; RMSE is in thousands of dollars.
Worked Example: A Fixed Classification Threshold
A hypothetical claims review team has finalized a classifier that flags claims for additional review. Its probability threshold was fixed using development data, before a separate test set was examined. On 40 test claims, its results are 8 true positives, 2 false negatives, 4 false positives, and 26 true negatives. Calculate accuracy, sensitivity, and specificity. Do not change the threshold based on these results.
Step 1: Check the total. The four counts add to \(8+2+4+26=40\), matching the stated test-set size.
Step 2: Calculate accuracy. There are \(8+26=34\) correct classifications. Accuracy is \(34/40=0.85\), or 85%.
Step 3: Calculate sensitivity. There are \(8+2=10\) actually positive claims. Sensitivity is the proportion of these correctly flagged: \(8/10=0.80\), or 80%.
Step 4: Calculate specificity. There are \(26+4=30\) actually negative claims. Specificity is \(26/30=0.8667\), or 86.67% to two decimal places.
Interpretation and check. The classifier correctly labels 85% of the test claims overall, detects 80% of the positive claims, and correctly leaves 86.67% of the negative claims unflagged. The denominators differ: sensitivity uses actual positives, while specificity uses actual negatives. A common mistake is to change the threshold after seeing the test results to improve sensitivity. That would make the reported test performance reflect a decision made with test information, rather than an independent final assessment of the fixed classifier.
Worked Example: Applying a Pre-Specified Performance Gate
A hypothetical reserving team has fixed a model and its prediction procedure before opening a test set. The team also specified in advance that the model would meet its performance criterion only if test RMSE did not exceed 2.5 thousand dollars. On four test claims, observed amounts are 8, 10, 14, and 18 thousand dollars; predictions are 10, 12, 11, and 22 thousand dollars. Calculate RMSE and decide whether the pre-specified criterion is met.
Step 1: Find the residuals. Observed minus predicted gives \(-2,-2,3,-4\), in thousands of dollars.
Step 2: Square and average. The squared residuals are \(4,4,9,16\), with sum \(33\). Thus test MSE is \(33/4=8.25\) thousand-dollars squared.
Step 3: Take the square root and compare. Test RMSE is \(\sqrt{8.25}=2.8723\) thousand dollars, rounded to four decimal places. Since \(2.8723>2.5\), the model does not meet the criterion.
Interpretation and check. The result supports the pre-specified conclusion that this model does not satisfy the stated performance gate. It does not justify changing the model and then presenting a new score on these same four claims as an untouched final evaluation. A common mistake is to treat any post-test model change as harmless because the original criterion was fixed. The criterion can support a pre-planned evaluation or deployment decision, but it does not keep the test set independent after its results are used to revise the model.
Use the Result as an Evaluation, Not a Tuning Signal
A final test assessment is most informative when the target population, the prediction task, and the performance metric are clear. The observations should represent the cases to which the result is intended to apply. If the test population differs materially from the intended future population, the reported estimate may answer the wrong question even if no test observation influenced model development.
The metric should also match the prediction goal. For example, squared-error measures emphasize large errors more than absolute-error measures; classification accuracy does not by itself describe the costs of false positives and false negatives. A metric selected after looking at test results can itself become part of the adaptation. Decide what to report and how to interpret it before evaluating the test set.
A test result can inform a pre-specified decision, such as whether a finalized procedure meets a stated minimum performance requirement. That is different from using the result to choose among models, thresholds, or transformations. If the gate is not met, the appropriate conclusion is that the model did not meet the stated requirement. Any subsequent development creates a revised procedure that needs a new independent evaluation for a fresh final performance claim.
Common Mistakes
- Treating the test set as another validation set. Validation results can guide development; the final test result should assess the procedure after those choices are complete.
- Reporting a tuned test score as independent. Selecting the best-performing test-set option uses test information to make the selection, so the resulting score is not an untouched evaluation.
- Ignoring the metric’s units or denominator. MSE has squared outcome units, RMSE has the original units, and classification rates can use different denominators.
- Overstating precision. A test estimate is subject to sampling variation. A small test set does not provide a guarantee of future performance.
- Changing the target after seeing the result. Reporting only the metric that looks most favorable after evaluation makes the assessment less neutral.
Check Your Understanding
For calculation questions, retain the units of the outcome and round probabilities or rates to four decimal places where needed.
- A finalized regression procedure has test residuals \(-1,2,-2\), with the outcome measured in thousands of dollars. Find test MSE and RMSE. (Answer: MSE \(=3\) thousand-dollars squared; RMSE \(=\sqrt{3}=1.7321\) thousand dollars.)
- A fixed classifier has 12 true positives, 3 false negatives, 5 false positives, and 30 true negatives on a test set. Find accuracy, sensitivity, and specificity. (Answer: accuracy \(=0.8400\), sensitivity \(=0.8000\), specificity \(=0.8571\).)
- A model has a test-set RMSE of 2.3. After inspecting that result, an analyst changes one tuning setting and reports the score of the revised model on the same test observations as a final independent estimate. Is that interpretation justified? (Answer: No; the revised procedure was changed using test information, so the same test set is no longer an independent final evaluation of it.)
- On four test observations, actual values are 6, 9, 12, and 15, while predictions are 7, 8, 10, and 17. Find test MSE and RMSE. (Answer: MSE \(=2.5\); RMSE \(=\sqrt{2.5}=1.5811\), in the outcome’s units.)
- Before evaluation, a team specifies that test RMSE must be at most 2.0 to pass. The observed test MSE is 5.29. Does the procedure pass, and should the team lower the RMSE by changing the model and reusing this test set for a fresh final score? (Answer: It does not pass because RMSE \(=\sqrt{5.29}=2.3>2.0\); it should not treat a revised model’s score on the same test set as a fresh independent final evaluation.)