From a Data Set to a Holdout Evaluation
A regression model can fit the data used to estimate it well and still predict new observations poorly. To assess prediction on cases not used to fit the model, set aside some observations before fitting. The set-aside cases form the test set; the remaining cases form the training set. A holdout evaluation fits on the training set and scores predictions on the untouched test set.
The separation is essential: if a test outcome influences the fitted model or a choice made about it, the resulting score is no longer an evaluation on untouched cases. A holdout score is an estimate of predictive performance for a model fitted to this particular training set, when predictions are made for cases from the intended setting.
For a quantitative outcome, common summaries use the prediction error \(e_i=y_i-\hat{y}_i\). The sign records whether the prediction was too low or too high; error measures based on absolute values or squares summarize its size.
Constructing the Split and Fitting the Model
A straightforward holdout evaluation follows a fixed sequence. Decide which observations the prediction task concerns, randomly assign cases to the two sets, and keep the test outcomes aside while fitting. The split should be made at the level of the unit for which predictions are intended. If the intended task is to predict outcomes for new people, for example, randomly separating individual records may be unsuitable when several records belong to the same person.
A random split is useful when the cases are exchangeable for the intended task: in particular, test cases should be drawn from the same kind of setting as the cases to which the reported performance is meant to apply. Random allocation alone cannot make a test set representative of a different population or a different prediction setting.
Identify the outcome, predictors, and regression fitting method to be evaluated. Check that the training data will allow that method to be fitted; for a simple least-squares line with an intercept, for example, training predictor values must not all be identical.
Assign each case to exactly one set. Keep the sets disjoint, and do not use test outcomes to make the assignment.
Estimate the regression coefficients using only training cases. Calculate test predictions from their predictor values and this fitted model; do not refit using test outcomes.
Compare each test prediction with its observed outcome, calculate the chosen measure or measures, and state their units and the cases to which the evaluation applies.
The fitting method and any data-dependent choices that affect the model must be determined without using test outcomes. For example, fitting a line on the training set and then changing its coefficients after looking at test errors means the reported score is not an untouched holdout evaluation. The same principle applies to using test outcomes to select predictors or decide between alternative fitted models.
Worked Example: Fit a Line and Score Two Held-Out Cases
Suppose six hypothetical observations relate study time \(x\), in hours, to a quiz score \(y\), in points. A random assignment puts the first four observations in the training set and the last two in the test set.
| Case | Study time \(x\) (hours) | Quiz score \(y\) (points) | Set |
|---|---|---|---|
| 1 | 1 | 2 | Training |
| 2 | 2 | 3 | Training |
| 3 | 3 | 5 | Training |
| 4 | 4 | 4 | Training |
| 5 | 5 | 6 | Test |
| 6 | 6 | 7 | Test |
Fit the least-squares line with an intercept using only cases 1–4. Their means are \(\bar{x}=2.5\) hours and \(\bar{y}=3.5\) points. The slope is
The intercept is \(b_0=\bar{y}-b_1\bar{x}=3.5-(0.8)(2.5)=1.5\) points. Thus the fitted model is \(\hat{y}=1.5+0.8x\). The training predictor values include more than one distinct value, so the line can be fitted.
Apply that unchanged line to the two test predictor values. At \(x=5\), the prediction is \(1.5+(0.8)(5)=5.5\) points, giving error \(6-5.5=0.5\) points. At \(x=6\), the prediction is \(1.5+(0.8)(6)=6.3\) points, giving error \(7-6.3=0.7\) points. Therefore,
and \(\mathrm{RMSE}=\sqrt{0.37}=0.608\) points. For these two held-out cases, the average absolute prediction error is 0.6 quiz points. This result describes this evaluation; it does not establish that future prediction errors will have exactly that size.
Why the Test Average Is an Evaluation
The test-set measure averages losses on cases that did not determine the fitted coefficients. To see why that matters, consider a fitted model as fixed and suppose the test cases are independent draws from the prediction setting. For a loss function \(L(y,\hat{y})\), define the model's expected prediction loss as the mean loss for a new case from that setting. Each test loss then has this same expectation. By linearity of expectation, the average of \(m\) test losses has that expectation too:
This justification depends on the test cases representing the intended prediction setting. It explains the score as an estimate of average loss for the fitted model, not as a guarantee about every future case. It also does not mean the estimate is exact: a different random test set could produce a different score. A single holdout evaluation gives one estimate for one fitted model and one split.
For ordinary least-squares point predictions, calculating a test MAE or MSE does not require normally distributed errors. The test outcomes are compared with predictions; the calculation itself is an average of observed losses. This does not remove the need for a suitable sampling design, a meaningful target population, or an appropriate fitted regression model.
Worked Example: Calculate Several Scores From Test Predictions
In a hypothetical evaluation of a model for daily water use, the regression was fitted using training cases only. On three untouched test cases, the observed use and predictions are as follows. The unit is litres per household per day.
| Test case | Observed \(y_i\) (litres) | Predicted \(\hat{y}_i\) (litres) | Error \(y_i-\hat{y}_i\) (litres) |
|---|---|---|---|
| 1 | 10 | 9 | 1 |
| 2 | 12 | 14 | -2 |
| 3 | 15 | 14 | 1 |
The absolute errors are 1, 2, and 1 litres, so \(\mathrm{MAE}=(1+2+1)/3=1.333\) litres per household per day. The squared errors are 1, 4, and 1 litres squared, giving \(\mathrm{MSE}=(1+4+1)/3=2\) litres squared. Thus \(\mathrm{RMSE}=\sqrt{2}=1.414\) litres per household per day.
MAE and RMSE are on the original outcome scale; MSE is not. The second case contributes more to MSE because its error is squared: an error of 2 contributes 4, while an error of 1 contributes 1. The measures therefore summarize the same test predictions in different ways.
Common Pitfall: Letting the Test Set Change the Model
A common mistake is to inspect test errors, adjust the regression, and then report the improved error on those same cases as if they were still untouched. For instance, if one test case has a large residual, a person might add a predictor or alter the model until that case is predicted more closely. The test outcomes have now influenced the fitted procedure. The score on those cases no longer provides the same independent check of predictions as before.
The correction is to preserve the test outcomes for evaluation: fit and make model-related choices without them, calculate predictions on the test cases, and report the chosen test measure. If test results are used to alter the procedure, the score from that test set should not be presented as an untouched evaluation of the altered procedure.
Check Your Understanding
Answer each question using the definitions and procedures in this tutorial.
- What condition must the training and test index sets satisfy, and why must the test outcomes be excluded from fitting?
- A regression gives test errors of \(-1\), \(2\), and \(1\) units. Calculate the MAE and MSE.
- Why does RMSE have the same units as the outcome, while MSE has squared units?
- Does a low test-set MAE guarantee small errors for every future case? Explain.
- A model is changed after its test errors are inspected. Can its score on that same set still be described as an untouched holdout evaluation of the changed model? Why or why not?