Why Separate Data Roles Matter
A regression procedure can fit the observed data closely and still predict new outcomes poorly. To assess prediction responsibly, it is not enough to calculate an error score: we must also ask which data were used to make the decisions that produced the predictions? The roles of training, validation, and test data answer that question.
The three roles form a workflow. Training data are used to fit candidate procedures. Validation data help choose among those candidates or tune their settings. Test data are reserved for evaluating the procedure after those choices are complete. The distinction is about how data are used, not about different kinds of observations.
For example, if the outcome is a patient's length of stay and predictors include age and admission measurements, the training data can be used to estimate a regression model. Validation data can help decide whether to use one predictor set or another. Once that decision is fixed, test data can show how the selected procedure predicts observations that did not guide its construction.
A fair holdout evaluation also requires a sensible relationship between the held-out observations and the intended prediction task. In particular, the evaluation data should represent the target population or setting, and the way observations are separated must suit the data structure. A test score from one setting does not automatically describe performance in a different population, period, or measurement process.
What Each Set Can Support
The roles are easiest to maintain by making an explicit list of decisions. A regression workflow may involve choosing predictors, selecting a functional form, setting a tuning value, or choosing among several candidate procedures. Training data support fitting each candidate. Validation data support choices among candidates. Test data support reporting a final assessment—not making another choice.
| Data role | Permitted use | What it should not decide |
|---|---|---|
| Training | Estimate coefficients and fit candidate procedures | Final performance on genuinely unused observations |
| Validation | Compare candidates and select settings or a procedure | A final, untouched performance assessment after selection |
| Test | Evaluate the procedure after the choices are fixed | Which procedure, setting, or result to choose next |
In practice, decisions often come in stages. For instance, a researcher might fit several regression forms on the training data, compare their validation errors, and select one. If the validation results prompt a new candidate or a change to the analysis, that is still a selection decision informed by validation. The test set remains unavailable for that decision.
After selection, a practitioner may sometimes refit the chosen procedure using both training and validation observations, then evaluate that refitted procedure on the test data. This can use more observations for fitting, but it changes what is being evaluated: the test score now assesses the refitted procedure, rather than the version fitted only to training data. In either case, the test outcomes must not guide the selection or subsequent adjustment if the score is to serve as the final holdout evaluation.
Worked Example: Choosing a Regression Procedure
Suppose a hypothetical clinic network has 120 records, each with a patient's age, a baseline symptom score, and length of stay in days. Before analysis, the records are separated into 72 training, 24 validation, and 24 test observations. The purpose is to predict length of stay for patients in the same intended setting. The split is assumed to leave each set representative of that setting and to avoid overlap between sets.
Two candidate procedures are specified in advance. Procedure A is a linear regression using age and symptom score. Procedure B uses the same predictors and also includes a squared symptom-score term. Both are fitted using only the 72 training records. Their mean squared prediction errors on the validation set are as follows:
| Candidate procedure | Validation MSE (days squared) |
|---|---|
| A: linear predictor terms | 5.8 |
| B: includes a squared symptom-score term | 4.9 |
Because the validation MSE for B is lower, the analyst selects B, provided that this comparison was the planned basis for choosing between the two. The validation observations have supported a choice; they have not provided a final, independent evaluation of the selected procedure, because their error scores were used to select it.
The analyst now uses the selected procedure to produce predictions for the 24 test records and calculates a test error. That test result is reserved for reporting the performance assessment. It cannot be used to switch back to A, add another predictor, or adjust the procedure and then still be described as an untouched final evaluation. Those changes would make the test results part of the selection process.
Calculating a Test-Set Error
For a quantitative outcome, a common summary is the mean squared error (MSE). If \(y_i\) is the observed outcome and \(\hat{y}_i\) is the prediction for test observation \(i\), then the test MSE is the average of the squared prediction errors. Its square root, the root mean squared error (RMSE), is in the original outcome units.
The formulas describe how to summarize the test predictions; they do not by themselves make the test set appropriate. The procedure must already be fixed, the test outcomes must not have informed its construction, and the data should match the prediction setting of interest. The score summarizes errors in this particular test set and is an estimate of predictive performance, not a guarantee about every future outcome.
Worked Example: Computing a Final Test RMSE
A hypothetical selected regression procedure predicts travel time to discharge, measured in hours, for four test cases. Assume these four cases were not used to fit the procedure or choose it, and that the procedure and its predictions were fixed before their outcomes were examined.
| Test case | Observed hours, \(y_i\) | Predicted hours, \(\hat{y}_i\) | Error, \(y_i-\hat{y}_i\) | Squared error |
|---|---|---|---|---|
| 1 | 10 | 11 | -1 | 1 |
| 2 | 12 | 11 | 1 | 1 |
| 3 | 15 | 14 | 1 | 1 |
| 4 | 17 | 19 | -2 | 4 |
There are \(n_{\mathrm{test}}=4\) cases, and the sum of squared errors is \(1+1+1+4=7\) hours squared. Therefore,
For these four held-out cases, the average squared prediction error is 1.75 hours squared, and the RMSE is about 1.323 hours. The RMSE gives a summary on the scale of travel time, but it does not mean that every prediction is within 1.323 hours of its observed value. Nor does this small example alone establish performance for all patients or settings.
Why the Test Set Must Stay Out of Selection
The issue is not that a test set becomes mathematically unusable the moment someone sees a number. The issue is that decisions based on its results make the reported score optimistic as an assessment of the final selected procedure. A simple argument shows why.
Suppose two procedures have the same true prediction error \(R\), but their test-based error estimates contain random deviations \(e_1\) and \(e_2\), each with mean zero. Thus the estimates are \(R+e_1\) and \(R+e_2\). If the analyst reports whichever test error is smaller, the reported value is \(\min(R+e_1,R+e_2)\). Since the minimum of two numbers is no greater than their average,
Taking expectations, the mean of the right side is \(R\), because \(E(e_1)=E(e_2)=0\). Therefore, the expected smaller estimate is no greater than \(R\). In other words, choosing the better-looking result introduces a downward tendency even though neither estimate was biased before selection. Repeatedly trying procedures against the test set creates the same kind of problem: the reported winner has benefited from chance features of those particular test observations.
This argument explains the logic of the roles. Validation data can be used for selection, with the understanding that their best observed score is affected by that selection. The untouched test data provide a separate evaluation after the chosen procedure is fixed. Even then, the test score is an estimate based on a finite sample, and its usefulness depends on how well the test observations represent the intended prediction task.
Worked Example: When a Test Score Becomes a Selection Score
Imagine that an analyst fits three candidate regressions on training data and uses validation data to select Procedure C. The analyst then calculates test RMSE values of 2.4, 2.1, and 2.6 days for Procedures A, B, and C. At this point, the test set could provide a final assessment of C: its test RMSE is 2.6 days, assuming the procedure was fixed before these test outcomes were examined.
But suppose the analyst instead chooses B because its test RMSE is lowest, then reports 2.1 days as the final test performance. The test set has now supported a procedure choice. The reported score for B is selected partly because it happened to perform well on these particular test cases, so it no longer has the same status as an untouched final assessment.
A valid correction is to keep C as the selected procedure if validation determined that choice, and report C's 2.6-day test RMSE as its test result. If the analyst wants to select B instead, that is a new modeling decision; an independent final evaluation would require data that have not informed this new selection. The test results should not be used to make the choice and then described as though they had not been consulted.
A Practical Sequence and Common Mistake
A clear workflow can be recorded before calculating any scores:
State what will be fitted and which choices will be compared.
Estimate the coefficients for each candidate using training observations.
Apply the stated comparison rule and fix the selected procedure and settings.
Generate predictions and calculate the planned performance summary without using the result to revise the choice.
A common mistake is to say, “We used the test set only to compare the final two models, so it is still a test set.” The correction is that comparing models is selection, regardless of whether there are two candidates or twenty. Once a test result helps decide which model to keep, it has helped construct the selected procedure and cannot also provide an untouched final assessment of that selection.
Another important distinction is between an evaluation of a procedure and a guarantee about future performance. A low test RMSE is evidence about prediction errors in a relevant held-out sample; it does not establish that future errors will be identical, that all subgroups have the same accuracy, or that predictions are causally meaningful. Interpret the score in its units and in the population or setting represented by the test data.
Check Your Understanding
Use the roles of the three data sets to answer the following questions.
- A regression coefficient is estimated from the training observations. Which role does that use represent, and why?
- An analyst compares linear and quadratic regression using their validation RMSE values. What decision can those results support, and why are they not a final independent evaluation?
- A selected procedure has test MSE \(=9\) units squared. What is its test RMSE, and what are the units of that RMSE?
- An analyst chooses whichever of five candidate models has the smallest test error. Explain why that score should not be described as an untouched final assessment.
- After validation selects a procedure, an analyst refits it using training and validation observations before calculating test error. What procedure does the test score assess, and what must remain true for the test data to retain their evaluation role?