Why the Final Test Set Needs Protection
A test set is intended to answer a specific question: How well does this selected modeling procedure perform on data it has not seen? It can answer that question only if its outcomes have not influenced the selection or adjustment of the procedure. If a team repeatedly checks test scores while changing the model, those scores become part of model development rather than an independent final evaluation.
This distinction matters even when every calculation is correct. A test score can be computed without error and still be a misleading estimate because the model was chosen partly for having performed well on that test set. The issue is not a particular algorithm or metric; it is the role the data played in decisions.
The training data are used to fit model parameters. Validation data or cross-validation results support choices among candidate procedures during development. The test set is held apart until those choices are complete. These roles are defined by how the data are used, not by the names assigned to columns or files.
In the simplest setting, suppose a fixed prediction rule \(f\) is evaluated on independent observations \((X_i,Y_i)\) drawn from the target population. For squared-error loss, its generalization risk is \(R(f)=E[(Y-f(X))^2]\). The test-set mean squared error estimates this risk. This simple interpretation relies on the test observations representing the target setting and on the predictions being produced by a rule fixed without their outcomes. When observations are dependent, sampled by group, or ordered in time, the evaluation design must reflect that structure; an ordinary random split may not be appropriate.
For a fixed \(f\), if the test observations are independent draws from the target population and have finite squared error, each squared error has expectation \(R(f)\). By linearity of expectation,
This is the basic reason a held-out test mean squared error can estimate the risk of a fixed procedure. The condition that \(f\) is fixed independently of the test outcomes is essential: it does not hold if those outcomes helped select \(f\).
What Must Stay Out of Test-Set Decisions
A useful rule is that any choice influenced by test outcomes is a development choice. The test set should not decide which variables to include, which transformation to use, which tuning settings to retain, or which candidate model to report. Nor should it be used to adjust a prediction rule after inspecting its results. These choices can be made with training data and validation or cross-validation results instead.
The earlier material on fitting preprocessing only on training data addresses a related but distinct risk: information from held-out observations can leak into the fitted preprocessing itself. Protecting the final test set also means not using its results to guide decisions. A carefully fitted pipeline can still have an invalid final estimate if its designer repeatedly checks test performance and revises the pipeline in response.
Define the target population and a suitable split design. Keep test outcomes out of model development. For time-ordered or grouped observations, respect the relevant structure when forming partitions.
Fit candidate procedures on training data and use validation data or cross-validation to compare choices. Keep preprocessing and other learned steps within the development process.
Record the selected model, fitted transformations, tuning choices, and any rule that converts predictions into the reported output. Do not revise these based on test results.
Apply the frozen procedure to the test predictors, calculate the pre-specified performance measure using the test outcomes, and report the result with its limitations.
“Evaluate once” is a practical safeguard rather than a claim that a test score can literally be calculated only once. The key is that the score must not become a basis for selecting or improving the model while still being presented as an untouched final estimate. A planned calculation, audit, or correction of a clerical error does not automatically invalidate the set; a modeling decision guided by the observed test result does.
Worked Examples
Worked Example: Calculating a Preserved Test-Set RMSE
A hypothetical insurer develops a model to predict the administrative cost, in dollars, of a policy change. The development team uses training data and cross-validation to select a procedure, then freezes it before opening a five-record test set. The observed residuals, defined as actual cost minus predicted cost, are \(1,-2,2,-1,\) and \(0\) dollars. Calculate the test RMSE and state what the result estimates.
Method. Calculate the mean squared error by squaring each residual, adding the squared values, and dividing by the number of test records. Then take the square root. The important protection here is that the test data did not select the procedure.
Step 1: Square the residuals. The squared residuals are \(1^2=1\), \((-2)^2=4\), \(2^2=4\), \((-1)^2=1\), and \(0^2=0\). Their sum is \(1+4+4+1+0=10\) dollars squared.
Step 2: Calculate the test MSE. There are five test records, so the MSE is \(10/5=2\) dollars squared.
Step 3: Take the square root. The RMSE is \(\sqrt{2}=1.414213\ldots\) dollars, or $1.41 rounded to the nearest cent.
Check and interpretation. Squaring $1.41 gives about 1.9881, close to the unrounded MSE of 2; the small difference is due to rounding. Under the stated representativeness and independence conditions, this test score estimates the squared-error performance of the frozen procedure on the target population. Five records provide a very small evaluation sample, so the estimate may be unstable even though the arithmetic is correct.
Worked Example: When Comparing Models Uses Up the Test Set
A team has already fit two hypothetical cost-prediction models. After looking at the same four test outcomes, it calculates residuals of \(1,2,3,4\) dollars for Model A and \(0,1,2,3\) dollars for Model B. The team proposes to select the model with the smaller test RMSE and report that score as its final estimate. Calculate both scores and assess the proposal.
Method. Calculate each model’s test RMSE, then distinguish the numerical comparison from the validity of the resulting estimate. The common mistake is to treat the minimum among test scores as if that score came from a model selected independently of the test data.
Model A. Its squared residuals are \(1,4,9,\) and \(16\), with sum \(1+4+9+16=30\). Dividing by four gives MSE \(30/4=7.5\) dollars squared. Thus its RMSE is \(\sqrt{7.5}=2.738612\ldots\), or $2.74 to the nearest cent.
Model B. Its squared residuals are \(0,1,4,\) and \(9\), with sum \(0+1+4+9=14\). Dividing by four gives MSE \(14/4=3.5\) dollars squared. Its RMSE is \(\sqrt{3.5}=1.870828\ldots\), or $1.87 to the nearest cent.
Decision. Model B has the lower observed test RMSE, so the team would select B under its proposed rule. But because the test results determined that choice, the $1.87 score is no longer a final estimate from an untouched test set. It is a result used for model comparison and is vulnerable to optimism.
Correction. The team should select between A and B using development data, validation data, or cross-validation, then evaluate the chosen procedure on a test set not used in that comparison. If the team has already made its choice using this test set, it should describe the score as part of development rather than as an untouched final result, and use a genuinely new evaluation set for a new final assessment if one is available.
Why Repeated Test Comparisons Become Optimistic
The mechanism is selection. Even when each candidate’s test score contains random sampling error, the candidate with the most favorable observed score is likely to be one whose error happened to be favorable on that particular test set. This is the same general issue that arises when validation results are reused to make many choices; the earlier material on why reusing validation results causes optimism gives the broader context.
A short argument makes the direction clear. Suppose, just for illustration, that \(J\) candidate procedures have the same true risk \(R\), and their test estimates can be written as \(\widehat{R}_j=R+\varepsilon_j\), where each estimation error has expectation zero. If the team selects the candidate with the smallest test estimate, its reported estimate is
For every set of values, \(\min_j\varepsilon_j\leq \varepsilon_k\) for each \(k\). Taking expectations therefore gives \(E[\min_j\varepsilon_j]\leq E[\varepsilon_k]=0\). Consequently, \(E[\min_j\widehat{R}_j]\leq R\). If at least two estimates can differ and the minimum is genuinely selected, the inequality is typically strict. Thus the selected minimum tends to understate the common true risk. This argument illustrates the selection effect; it does not claim that the candidate risks are actually equal or that every comparison has the same size of bias.
When candidate risks differ, the selected model may genuinely be better. However, selecting it based on test outcomes still makes the same test score an optimistic basis for reporting its performance: the selection favored favorable test noise as well as any genuine improvement. The solution is not to avoid comparison, but to conduct it with development data and reserve the final test set for the selected procedure.
After the Test Result: What Can and Cannot Be Done
Once the planned final score is calculated, it is reasonable to report it with the split design, sample size, metric, and relevant limitations. It is not reasonable to inspect a poor score, alter features or tuning settings, and then report the improved score on the same set as if it were independent. The changed procedure has now been influenced by the test outcomes.
If test results prompt a substantive modeling change, treat the test set as development information from that point onward. Further development can use the available data, but a new independent test set is needed to obtain a fresh final estimate. If no new test data can be obtained, report the limitation plainly rather than implying that the original test result remains untouched. A separate test set is not always practical, particularly for small data; in that case, a validation design appropriate to the data may be preferable to repeatedly reusing a nominal “test” set.
Worked Example: A Time-Ordered Final Evaluation
A hypothetical service provider has 1,000 monthly records ordered by date. The team assigns the first 800 records to model development and the next 200 records to a final test period. Within the 800 development records, cross-validation selects a procedure. The team then fits that selected procedure on all 800 development records and freezes it before evaluating the 200 later records. The sum of squared dollar residuals on the test period is $5,000 squared. Calculate the test RMSE and explain the role of the later records.
Method. Because the records are time-ordered and the intended use is prediction in a later period, the stated chronological evaluation is the relevant design. The later test outcomes must not influence the cross-validation choices or the final fit.
Step 1: Calculate test MSE. The test period contains 200 records, so \(5{,}000/200=25\) dollars squared.
Step 2: Calculate test RMSE. The RMSE is \(\sqrt{25}=5\) dollars.
Step 3: Interpret the result. The $5 RMSE describes performance on the stated later period for the procedure fitted on the first 800 records. It does not prove that future performance will always have this value; changes in the population or operating conditions can alter performance.
Check. Squaring the $5 RMSE returns the MSE of $25 squared, and multiplying by 200 gives the stated squared-error total of $5,000. The common mistake would be to revise the procedure after seeing this period’s results and still call the resulting score an untouched final evaluation.
Common Mistakes and Practical Safeguards
- Choosing the winner by test score. A test score used to compare candidates is no longer an untouched final score. Use development comparisons for selection.
- Making “small” changes after inspection. Changing a transformation, feature set, or tuning choice because of test results is still test-guided development.
- Confusing an unused outcome column with an untouched test set. If the team inspected a test score and acted on it, the set has influenced decisions even if no code directly reads the outcome column.
- Assuming a protected set guarantees representativeness. A perfectly untouched test set can still be a poor basis for the intended target if its sampling or time period does not match that target.
- Reporting more precision than the design supports. Give the metric and test sample size, and avoid presenting a small-sample estimate as exact or universally stable.
Check Your Understanding
For each situation, identify the relevant calculation or data-role decision. Answers are included so you can check your reasoning.
- A hypothetical study has 900 eligible records. It assigns 75% to development and the remainder to a test set. How many records are in each partition, and may the team use the test outcomes to choose between two candidate models? (Answer: Development has 675 records and the test set has 225; no, use development-based validation or cross-validation for that comparison.)
- Two models have already been compared using the same test set, and the team selected the one with the smaller test RMSE. Is that RMSE still an untouched final estimate? State the appropriate interpretation. (Answer: No; the test set influenced model selection, so the score is a development comparison and may be optimistic.)
- A frozen model has four test residuals of \(2,-3,4,\) and \(-1\) dollars. Calculate its test MSE and RMSE, rounding RMSE to the nearest cent. (Answer: The squared residuals sum to \(4+9+16+1=30\); MSE is \(30/4=7.5\) dollars squared, and RMSE is \(\sqrt{7.5}\approx \$2.74\).)
- Three equally performing candidate procedures have observed test-risk estimates of 12.6, 13.1, and 13.6, while their common true risk in a simplified illustration is 13.0. Which estimate would be selected by minimizing test risk, and what does this show? (Answer: 12.6 would be selected, which is 0.4 below the common true risk in this illustration; selection can favor favorable estimation noise.)
- A team reports its frozen model’s test score, then changes the feature set after inspecting errors on the test records. What status does the set now have, and what is needed for a new final estimate? (Answer: It has become development information for the changed procedure; a new independent test set is needed for a fresh final estimate.)