Why Tuning Needs Careful Evaluation
A regression model often involves choices beyond which variables to include. For example, a fitting method might have a penalty-strength setting, a choice among several model forms, or a setting that controls model complexity. These choices can affect predictions on new observations. Selecting settings by looking only at training performance is not enough: a more flexible setting can fit the training data closely without predicting new outcomes as well.
Validation data or cross-validation can compare candidate settings using observations not used to fit the corresponding model. But the score used to choose a setting has a special role: it guides the choice. If that score is then reported as though it were an independent, final assessment, it can be too optimistic. A reliable workflow separates choosing the model settings from assessing the chosen procedure.
A Basic Separation of Roles
Suppose the available data are divided into a development set and a test set. The development set is used to fit candidate models and choose among tuning settings. That choice may use a separate validation set within the development data, or cross-validation on the development data. The test set is not used to choose predictors, settings, preprocessing steps, or any other part of the procedure. After the choices are settled, predictions for the test observations can be compared with their observed outcomes to estimate predictive performance.
For a loss where smaller values indicate better predictions, such as mean squared error (MSE), the selected setting is the one with the smallest validation or cross-validation score. If the score is defined so that larger is better, the direction of selection is reversed. Scores are estimates: different validation samples or fold assignments can produce different results, and small score differences need not indicate a real performance difference.
Cross-validation provides several validation evaluations within the development data. In each fold, a model is fitted on the training portion and scored on the held-out portion; the fold scores are then combined. To compare tuning settings fairly, use the same fold assignments for each setting. Any preprocessing that learns from data must be fitted using only the training portion of each fold, as in the earlier material on fitting preprocessing steps inside each fold.
Worked Example: Selecting a Setting With a Validation Set
A hypothetical regression analysis predicts a laboratory measurement in milligrams per litre. Four validation observations have measured outcomes \(10, 12, 14,\) and \(16\) mg/L. Three candidate settings produce the predictions below. For this example, compare settings using validation MSE, measured in \((\text{mg/L})^2\).
| Setting | Validation predictions (mg/L) |
|---|---|
| A | 9, 13, 13, 17 |
| B | 10, 11, 15, 16 |
| C | 8, 12, 14, 18 |
For setting A, the prediction errors (observed minus predicted) are \(1,-1,1,-1\) mg/L. Thus
For setting B, the errors are \(0,1,-1,0\) mg/L, so
For setting C, the errors are \(2,0,0,-2\) mg/L, giving
Setting B has the smallest validation MSE of the three and is selected on this criterion. The result says that B performed best on these validation observations among these candidates. It does not establish that B will have the smallest error in every new sample. If a test set was reserved, it remains unused at this stage.
Choosing Settings With Cross-Validation
In cross-validation, each candidate setting is evaluated across the same folds of the development data. For a loss such as MSE, the cross-validation score is commonly the average of the fold scores. This gives each setting a score based on predictions for observations left out of the fit that produced those predictions. It does not make the score independent of tuning: if the score determines which setting is chosen, it has been used to make a decision.
Worked Example: Comparing Settings Across Four Folds
A hypothetical analysis uses four-fold cross-validation to tune a regression procedure for predicting concentration in micrograms per millilitre. The table gives the MSE on each held-out fold, in \((\text{micrograms/mL})^2\). The same four fold assignments were used for each setting.
| Setting | Fold 1 | Fold 2 | Fold 3 | Fold 4 |
|---|---|---|---|---|
| A | 8 | 10 | 9 | 11 |
| B | 7 | 8 | 9 | 8 |
| C | 12 | 7 | 10 | 9 |
For setting A, the average fold score is
The other averages are
Setting B has the lowest average cross-validation MSE, so it is selected from these three settings. Its score summarizes performance across the held-out folds, not an independent test of the final selection: the scores were examined to decide which setting to use. If a separate test set was reserved before tuning, it can provide the final assessment after the setting and other analysis choices have been fixed.
Why Searching Many Settings Can Make Scores Optimistic
A validation score varies partly because it is calculated on a finite set of observations. Even if several candidate settings have similar underlying predictive performance, chance variation can make one score look unusually good. Searching many candidates and selecting the smallest score tends to select not only a promising setting, but also one that benefited from favorable variation in the evaluation data.
The basic logic can be stated precisely. Suppose there are a fixed number of candidate settings \(1,\ldots,m\). Let \(R_j\) be the expected prediction loss for setting \(j\) on the target population, and let \(\widehat{R}_j\) be its validation estimate. Assume each estimate is unbiased for its corresponding expected loss, so \(E(\widehat{R}_j)=R_j\). The selected setting is the one with the smallest estimated loss. For every fixed candidate \(k\), the smallest observed estimate cannot exceed \(\widehat{R}_k\). Taking expectations gives the following result.
Therefore,
The minimum validation estimate is, on average, no greater than the smallest candidate's expected loss. Meanwhile, the expected true loss of the selected setting cannot be below the best candidate's expected loss. Thus the selected validation score tends to be optimistic as an assessment of the selected setting's predictive performance. The effect can be more pronounced when many candidates are searched or when the scores are noisy. This reasoning does not say that every selected score is too low, or that the setting with the smallest observed score must perform poorly; it explains why the selection score alone is not a neutral final assessment.
Worked Example: Repeatedly Adjusting a Model to One Validation Set
Suppose an analyst repeatedly changes a hypothetical regression model after inspecting its score on the same validation observations. Each adjustment is intended to improve validation MSE, so the analyst is effectively searching over an expanding set of candidates. The final model predicts outcomes for three validation observations of \(30,32,\) and \(34\) mg/L as \(28,33,\) and \(33\) mg/L.
The validation errors are \(2,-1,\) and \(1\) mg/L, so the reported score for the chosen model is
If these observations guided many successive changes, this value reflects both the model's predictions and the analyst's repeated adaptation to this particular validation set. To illustrate the separate final assessment, suppose an untouched test set has outcomes \(40,42,\) and \(44\) mg/L, and the now-fixed model predicts \(37,42,\) and \(41\) mg/L. Its test errors are \(3,0,\) and \(3\) mg/L, giving
The test MSE is higher in this hypothetical example. That difference is consistent with an overly favorable validation score, but the two scores could also differ because both sets are small and subject to sampling variation. A single example cannot determine why they differ. The key safeguard is that the test observations were not used to generate or choose the model; otherwise, the test score would also become part of the tuning process.
A Practical Workflow and Its Limits
Do this before tuning. Do not use test outcomes to choose predictors, settings, transformations, or preprocessing decisions.
Define the settings to compare and the score to use. Keep the fitting and preprocessing steps within the appropriate training portions.
Use validation data or cross-validation to compare candidates. Choose the setting according to the stated score and record the decision.
After the choices are fixed, use the test data for a final estimate of predictive performance. Do not return to tuning based on its score.
A common mistake is to reason, “The chosen setting had the best validation score, so that score is an unbiased estimate of its future performance.” The correction is that the score was used to select the setting; among noisy candidates, the smallest score tends to be unusually favorable. Use validation or cross-validation for selection, and reserve an untouched test set for a final assessment when one is available.
If the test result prompts changes to the model, the test set is no longer untouched: it has influenced a decision. Reporting that same test score as an independent assessment after those changes is not justified. Likewise, repeatedly checking a validation score while making adjustments gradually turns validation into a tuning resource rather than an independent assessment. Clear separation of roles makes the resulting performance claim more interpretable.
Check Your Understanding
Try these questions without looking back.
- In a tuning workflow, what is the difference between the role of validation data and the role of a test set?
- For a loss where smaller values are better, how would you choose among settings with cross-validation scores of 4.2, 3.8, and 4.0?
- Why can the minimum validation score become optimistic when many settings are compared?
- An analyst changes a model repeatedly after checking its validation score. Is that validation set still an independent assessment? Explain.
- Why should a test set not be used to choose a predictor or a tuning setting if its score is intended as a final assessment?