Why Reserve a Final Test Set?
Cross-validation helps compare candidate models and tune their settings, but it does not make the data used during that process independent of the chosen model. If the same observations guide model selection and then appear to assess the selected model, the reported performance can be optimistic. A final test set addresses this problem by setting aside observations before model development begins and keeping them out of every fitting and tuning decision.
The central rule is simple: use development data to build and select the modeling procedure; use the final test data only to assess the resulting procedure once. The test set is not another validation fold, a source of tuning feedback, or a place to decide which result to report. This distinction is useful when an actuarial analysis needs a final estimate of predictive performance on observations that did not influence model selection.
Separate the Data Before Model Development
A standard workflow first divides the available observations into development and test sets. The split should reflect the population and prediction task of interest. For independent observations drawn from one population, a random split may be reasonable. For repeated records from the same policyholder, patient, or business, splitting individual rows at random can put nearly identical information in both sets. A group-based split should instead keep each group entirely on one side. When the goal is to predict future outcomes, a time-ordered split is often more appropriate than a random split.
Once the split is made, cross-validation and all model choices stay within the development data. For example, when using \(K\)-fold cross-validation, each fold is formed from development observations only. For each candidate model, the fitting and validation steps occur within those folds; the test observations do not enter any fold. The test set remains closed while the analyst chooses the model family, tuning parameters, preprocessing rules, and loss function.
Preprocessing is part of model development. If the pipeline imputes missing values, centers or scales predictors, selects features, or estimates category groupings, those quantities must be learned from the relevant training portion of development data. During cross-validation, the preprocessing step is fitted on each fold's training portion and then applied to that fold's validation portion. After model selection, the fixed modeling procedure is applied to the test set without using its outcomes to alter the procedure.
Choose a split suited to the sampling structure and prediction goal. Record which observations are reserved, and do not use them to make modeling decisions.
Use development data for preprocessing, candidate comparisons, and cross-validation. Fit learned preprocessing separately within each cross-validation training fold.
Decide the model and tuning choices using development results, not test results. Apply the fixed procedure to obtain predictions for the reserved test observations.
Calculate the prespecified test loss or losses and report their interpretation and limitations. Do not use the results to make another round of model choices.
What Does a Test Score Estimate?
Let \(D\) denote the development data and let the modeling procedure, after using \(D\), produce a prediction rule \(\hat{f}_D\). Let \((X_j,Y_j)\), for \(j=1,\ldots,n_{\text{test}}\), be the test observations, and let \(L(Y_j,\hat{f}_D(X_j))\) be the loss for observation \(j\). The mean test loss is
The loss must match the prediction task: for example, squared error for a regression assessment or log loss for probabilistic classification. The test score estimates performance on new observations only to the extent that the test observations represent the target population and the loss is appropriate. A small test set can produce a noisy score, and a distribution shift between test observations and future observations can make the score unrepresentative.
The reason the test average is useful is that the model is fixed before the test outcomes are examined. Suppose, conditional on the development data, the test pairs are independent and identically distributed from the target population, and the loss has a finite expectation. Each test loss then has the same conditional expectation, say \(R_D\). Linearity of expectation gives
Thus, for a fixed procedure conditional on development data, the test average is an unbiased estimate of its expected loss under these assumptions. The result does not apply if the analyst repeatedly consults the test results and changes the model in response: the final procedure then depends on the test set, so the conditional argument no longer describes an independent assessment.
Worked Example: Allocating Observations Without Leaking Into Cross-Validation
A hypothetical insurer has 1,000 independent policy records for predicting annual claim cost. The analyst reserves 20% for a final test set and uses five-fold cross-validation on the remaining records to choose an elastic net tuning combination. Find the number of development and test observations, and the training and validation sizes in each cross-validation fold. State which observations may guide the tuning choice.
Method. Make the test split first, then partition only the development observations into five folds.
The test set has \(0.20(1{,}000)=200\) records. The development set therefore has \(1{,}000-200=800\) records. In five-fold cross-validation, each validation fold contains \(800/5=160\) records. The corresponding training portion contains \(800-160=640\) records.
The analyst can use the 800 development records, through their cross-validation results, to select the elastic net tuning combination. Each fit uses 640 development records and is evaluated on the remaining 160 development records. None of the 200 test records is included in these fits or validation folds. After the procedure is fixed, the reserved 200 records are used for the final assessment.
Check. The five validation folds contain \(5(160)=800\) development records in total, and each fold's training and validation portions sum to \(640+160=800\). The common mistake avoided here is running cross-validation on all 1,000 records after having labelled 200 of them as a test set.
Worked Assessment for a Regression Model
For a regression model, the mean absolute error (MAE) averages the absolute prediction errors, while the root mean squared error (RMSE) is the square root of the average squared errors. Both are expressed in the same units as the response. Squaring makes RMSE more sensitive than MAE to large residuals.
Worked Example: Calculating Final Test Errors in Thousands of Dollars
A fixed model, selected using development data, predicts annual claim costs for four reserved test policies. Its residuals, defined as actual cost minus predicted cost, are \(2,-2,0,4\), in thousands of dollars. Calculate the test MAE and RMSE. The analyst had specified both metrics before examining the test results.
Method. Apply the absolute-value and squared-error formulas to the four residuals. The residual magnitudes are \(2,2,0,4\), so their sum is \(8\). Therefore,
The squared residuals are \(2^2=4\), \((-2)^2=4\), \(0^2=0\), and \(4^2=16\). Their sum is \(4+4+0+16=24\), and their mean is \(24/4=6\). Hence,
Rounded to three decimal places, the RMSE is 2.449 thousand dollars. The MAE is exactly 2 thousand dollars. Check: RMSE exceeds MAE here, as expected because the squared-error metric gives relatively more weight to the residual of 4. The mistake avoided is reporting the mean squared error, \(6\), as the RMSE; RMSE requires taking the square root.
Classification Metrics and the Threshold Trap
For a binary outcome \(Y\in\{0,1\}\), a classifier may provide a predicted probability \(p_j\) of outcome 1. The mean log loss for test predictions is the average negative log probability assigned to the observed outcomes. For an observed value \(y_j\), the probability assigned to that outcome is \(p_j\) when \(y_j=1\), and \(1-p_j\) when \(y_j=0\). Probabilities should be strictly between 0 and 1 for finite log loss; implementations commonly protect against exact 0 or 1 predictions.
Accuracy is different: it first converts probabilities to predicted classes using a threshold, often 0.5, and then counts correct classifications. The threshold is a modeling decision and should be selected using development data if it is to be tuned. A threshold should not be chosen because it gives the best accuracy on the final test set.
Worked Example: Test Log Loss and Accuracy for a Fixed Classifier
A fixed classifier assigns probabilities \(0.9,0.8,0.4,0.2\) to four test outcomes \(1,0,1,0\), respectively. Using threshold 0.5, calculate the number classified correctly and the mean log loss. Round the log loss to four decimal places.
Method. First determine the assigned probability of each observed outcome. These are \(0.9\) for the first positive outcome, \(1-0.8=0.2\) for the second negative outcome, \(0.4\) for the third positive outcome, and \(1-0.2=0.8\) for the fourth negative outcome.
At threshold 0.5, the predicted classes are \(1,1,0,0\). Comparing with actual outcomes \(1,0,1,0\), the first and fourth predictions are correct. Thus accuracy is \(2/4=0.5\), or 50%.
The log loss is
The mean log loss is 0.7136 to four decimal places. Check: the low probability 0.2 assigned to the second observation's actual outcome contributes the largest loss. The common mistake avoided is using \(p_j\) as the probability of the observed outcome even when \(y_j=0\); for a negative outcome, the assigned probability is \(1-p_j\).
Common Mistakes and Practical Safeguards
The test set is valuable precisely because it is not part of the model-development feedback loop. Reporting a test score does not make it independent if the analyst has already used that score to revise the analysis. If the test results lead to a new feature, a different tuning value, a new threshold, or a choice between candidate models, the test set has become development data for that decision. A fresh test set would be needed for a genuinely independent final assessment.
- Using test information to preprocess. Computing a scaling mean or imputation value from both development and test observations lets the test predictors influence the fitted pipeline. Estimate such quantities from development training data and apply them to validation or test data.
- Repeatedly checking test performance. Trying several models and keeping the one with the best test score makes the test set part of selection. Use cross-validation within development data for those comparisons.
- Choosing the metric after seeing results. Selecting whichever metric makes the model look best weakens the assessment. Specify relevant losses in advance and interpret them in context.
- Using an unsuitable split. A random row split can leak information across repeated records or ignore the time direction of a forecasting task. Match the split design to the intended use.
- Overstating precision. A test score is an estimate, not a guarantee of future performance. Report the test-set size and note important sampling limitations.
A final test assessment complements, rather than replaces, development-stage validation. Cross-validation helps choose a procedure using the available development data; the test set gives one independent check after those choices are settled. The workflow is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus, particularly the distinction between tuning and performance assessment.
Check Your Understanding
Work each problem using the separation between development and final test data.
- A study has 500 independent records. The analyst reserves 20% for testing and applies five-fold cross-validation to the remaining records. How many records are in the test set, in development data, in each validation fold, and in each fold's training portion? (Answer: 100 test; 400 development; 80 per validation fold; 320 per training portion.)
- A fixed regression model has test residuals, in thousands of dollars, \(3,-1,2,-2\). Calculate MAE and RMSE, rounding RMSE to three decimals. (Answer: MAE \(=2\) thousand dollars; RMSE \(=\sqrt{4.5}\approx2.121\) thousand dollars.)
- A classifier gives probabilities \(0.7,0.4,0.2\) for test outcomes \(1,0,1\). At threshold 0.5, how many classifications are correct, and what is mean log loss to four decimals? (Answer: 2 of 3 correct; mean log loss \(=[-\ln(0.7)-\ln(0.6)-\ln(0.2)]/3\approx0.8256\).)
- A development set has predictor values \(2,4,6\), and a reserved test observation has predictor value \(100\). If standardization uses the development mean, what is that mean, and should the test value be included in calculating it? (Answer: development mean \(=4\); do not include the test value.)
- Two models have cross-validation mean losses of 12 and 14 on development data. The first model then has test loss 13, and the second has test loss 11. Which model should be described as selected if the selection rule was to choose the lower cross-validation loss, and what is the test-set trap? (Answer: The first model was selected by the stated rule; choosing the second because its test loss is lower would use the test set for model selection.)