Refitting a Selected Procedure Before Test Evaluation

Why Refit Before the Final Evaluation?

A data split creates separate jobs for different observations. Training data are used to fit candidate procedures; validation data help select among procedures or settings; and a test set is held back for a final evaluation. After selection, it is usually wasteful to leave the validation observations out of the final fit: they contain information about the same outcome and predictors, and they can be used to estimate the parameters of the already-selected procedure.

The order matters. First choose the procedure without using the test outcomes. Next refit that procedure using all training and validation observations. Only then use the test set to evaluate predictions from this final fit. The test set must remain separate from both selection and fitting if its evaluation is to provide an honest check of performance on new observations.

Definition: The development data are the combined training and validation observations, denoted \(D_{\mathrm{dev}}=D_{\mathrm{train}}\cup D_{\mathrm{val}}\). The selected procedure is the complete, already-chosen sequence for preparing the data and fitting a regression model, including any selected predictors, transformations, or tuning settings. Refitting means applying that fixed procedure anew to all of \(D_{\mathrm{dev}}\), estimating its data-dependent quantities from those observations. The test set \(D_{\mathrm{test}}\) is not used to select, prepare, or fit the procedure.

What Is Fixed, and What Is Re-estimated?

Selection chooses a procedure; refitting estimates the quantities that procedure requires. For example, if validation has selected a linear model with two specified predictors and a specified penalty setting, those choices are fixed for the final fit. The model coefficients, and any preprocessing quantities estimated from data, are recalculated from the combined training and validation observations. The test set supplies none of these quantities.

This distinction also applies when the procedure includes preprocessing. If a procedure imputes missing values using a variable's mean, scales a predictor using its mean and standard deviation, or selects variables as part of its defined fitting algorithm, those steps must be applied as specified using development data only. A quantity that is learned from data must not be calculated using test observations. Once the final fit is ready, the already-fitted preprocessing steps are applied to test predictors to produce predictions; they are not recalculated from the test set.

A procedure may itself contain a rule for selecting a setting from the development data. In that case, the rule must be followed using development data only, and the resulting fitted model is evaluated on the test set. If the setting was already chosen in an earlier selection stage, it remains fixed during the final refit. The key requirement is that the test set does not influence either the choice or the fit.

Worked Example: Recomputing Preprocessing on All Development Data

Suppose a hypothetical development set contains 60 training observations and 20 validation observations. The selected procedure imputes a missing value in a predictor using the mean of its observed development values, then fits a regression model. The predictor values sum to 300 across the training observations and to 120 across the validation observations; there are no missing values in this calculation. Find the mean used by the final refit.

The number of development observations is \(n_{\mathrm{dev}}=60+20=80\). The total predictor sum is \(300+120=420\), so the mean used for imputation in the final procedure is

\bar{x}_{\mathrm{dev}}=\frac{300+120}{60+20}=\frac{420}{80}=5.25.

The final regression fit is then estimated using all 80 development observations, with the selected procedure and its imputation rule. The mean 5.25 is not calculated from the test predictors. Even if test predictors contain missing values, they are handled using the imputation value learned from development data, not a new test-set mean.

Refit, Then Evaluate Once

Before the test data are opened, decide how predictions will be judged. Write the evaluation rule in advance and use it consistently. For example, if the chosen measure is the mean squared prediction error, define the loss for one observation as \(\ell(y,\hat{y})=(y-\hat{y})^2\), where \(y\) is the observed outcome and \(\hat{y}\) is the prediction. For a test set of size \(m\), the test score is

\widehat{R}_{\mathrm{test}}=\frac{1}{m}\sum_{i=1}^{m}\ell(y_i,\hat{y}_i).

Here \(\hat{y}_i\) is generated by the final model fitted on \(D_{\mathrm{dev}}\). The formula averages the preselected loss across the test observations; it does not refit the model. The score is expressed in squared outcome units for this particular loss. Other evaluation measures are possible, but the choice should be made without inspecting test outcomes.

Worked Example: Calculating a Final Test-Set Score

A hypothetical final regression model has been refitted on all development data. On four untouched test observations, the measured outcomes are 10, 12, 15, and 16 units, while the corresponding predictions are 11, 11, 14, and 18 units. Using the prespecified squared loss, calculate the average test loss.

For each observation, subtract the prediction from the outcome and square the difference. Thus the four losses are \((10-11)^2=1\), \((12-11)^2=1\), \((15-14)^2=1\), and \((16-18)^2=4\), in squared outcome units. Therefore,

\widehat{R}_{\mathrm{test}} =\frac{1}{4}\left[1+1+1+4\right] =\frac{7}{4} =1.75\ \text{units}^2.

The final model's average squared loss on these four test observations is 1.75 squared units. This is a result for the specified test set, not a guarantee that future prediction loss will equal 1.75. The small example is hypothetical and illustrates the calculation, not a reliable estimate for a real application.

Why a Held-Out Test Score Can Be Informative

A test score is useful when the test observations represent the kinds of new observations to which predictions will be applied, and when they have not influenced the procedure or the fitted model. A simple justification follows from averaging losses. Suppose the final fitted model is fixed, and the \(m\) test observations are independent draws from the same target population. Let \(R\) be the expected loss for a new observation from that population. Then each test loss has expectation \(R\), so

E\!\left(\widehat{R}_{\mathrm{test}}\mid \text{final fitted model}\right) =\frac{1}{m}\sum_{i=1}^{m}E\!\left(\ell(y_i,\hat{y}_i)\mid \text{final fitted model}\right) =\frac{1}{m}\sum_{i=1}^{m}R =R.

This shows that, under those conditions, the average test loss is an unbiased estimate of the expected loss for the fixed final fit. It does not say that the observed score must be close to \(R\): a small or variable test set can yield an imprecise estimate. Nor does it establish that the test observations represent a different population or a future time period. The interpretation depends on the sampling and deployment context.

The refitted model uses more observations than a model fitted on training data alone. Its coefficients may therefore differ from the earlier candidate's coefficients. The test score belongs to this final refitted model, not to the earlier training-only fit and not automatically to every future model that might be trained on a different sample.

Worked Example: Re-estimating a Regression Line Before Testing

Suppose a hypothetical selection step, completed without test outcomes, chose a simple linear regression procedure. Training observations are \((1,2)\), \((2,4)\), and \((3,5)\), where the first value is the predictor and the second is the outcome. The validation observation is \((4,9)\). Refit the selected linear procedure on all four development observations, then make a prediction for a test predictor value of 5.

For simple linear regression, the fitted slope is \(b_1=S_{xy}/S_{xx}\), where \(S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})\) and \(S_{xx}=\sum (x_i-\bar{x})^2\). The fitted intercept is \(b_0=\bar{y}-b_1\bar{x}\). These formulas are applied to the four development observations, not just the three training observations.

The means are \(\bar{x}=(1+2+3+4)/4=2.5\) and \(\bar{y}=(2+4+5+9)/4=5\). The centered sums are

S_{xx}=(-1.5)^2+(-0.5)^2+(0.5)^2+(1.5)^2=5,

and

S_{xy}=(-1.5)(-3)+(-0.5)(-1)+(0.5)(0)+(1.5)(4)=11.

Therefore the refitted slope is \(b_1=11/5=2.2\), and the intercept is \(b_0=5-(2.2)(2.5)=-0.5\). The final fitted line is \(\hat{y}=-0.5+2.2x\), so the prediction at the test predictor value \(x=5\) is \(\hat{y}=-0.5+(2.2)(5)=10.5\). This is a prediction from the development-data fit. The test outcome, when used for the preselected evaluation, does not alter the line.

Common Mistakes and a Practical Sequence

The most important pitfall is treating the test set as another validation set. If a test score looks disappointing and the analyst changes predictors, tuning settings, preprocessing, or the evaluation measure, then the test results have influenced the procedure. Reporting a new score on the same test data as if it were an untouched final evaluation no longer has the original interpretation. Choosing whichever candidate had the best test score creates the same problem: the test set has become part of selection.

A sound sequence keeps the roles explicit:

Complete selection without the test outcomes.
Use the development data to choose the procedure and any settings that the selection process is meant to determine.

Refit on all development data.
Apply the selected procedure to training and validation observations together, recalculating data-dependent preprocessing quantities and model parameters as specified.

Generate test predictions.
Apply the fitted development-data pipeline to test predictors. Do not update its preprocessing quantities or model parameters using test data.

Calculate the prespecified test score once.
Use the test outcomes only for the final evaluation, and report the result with its context and limitations.

If the test evaluation motivates further development, the existing test set has now served as feedback and should not be presented as an independent final evaluation of the revised procedure. A new untouched evaluation set would be needed for that claim. This does not mean that model development must stop; it means the description of what the test score can support must reflect how the data were used.

Key takeaway: Select without test data, refit the selected procedure on training plus validation data, and evaluate that final fit on the untouched test set using a measure chosen in advance. The test score is informative only to the extent that the test data remain independent of selection and fitting and represent the intended prediction setting.

Check Your Understanding

Answer each question using the distinction between selection, refitting, and final evaluation.

A selected procedure standardizes a predictor before fitting. Which observations should determine the mean and standard deviation used by the final refit?
After refitting a selected procedure on training and validation data, may its coefficients differ from those of the earlier training-only fit? Explain.
Why should the evaluation measure be specified before examining test outcomes?
State the assumptions that justify interpreting the average test loss as an unbiased estimate of expected loss for the fixed final model.
An analyst changes the predictor set after seeing an unexpectedly large test score, then reports a new score on the same test set. What is compromised, and why?