Using A Final Test Set After Penalty Tuning

Why Keep a Final Test Set?

Penalty tuning uses data to choose a model—for example, by comparing cross-validation losses across values of the penalty parameter $\lambda$. Those comparisons are useful for selecting a model, but they are not an independent final check: the selected value was favored, in part, because of how it performed on those data. An untouched test set provides a separate assessment of the selected model's predictive performance.

The word untouched matters. Before test outcomes are examined, the analyst should settle the model specification, selected penalty, preprocessing rules, and performance measures to be reported. If the test results lead to a change in any of those choices, the test set has become part of model development. Its results can still inform development, but no longer serve as an independent final assessment of the revised choice.

Definition: A final test set is a set of observations withheld from model fitting, preprocessing decisions, feature selection, penalty tuning, and other choices that determine the model. After those choices are fixed, the model produces predictions for the test observations and a prespecified performance measure is calculated. The test assessment describes that fixed model on data representative of the intended prediction population.

A typical division of roles is: use training data to fit candidate models, use cross-validation within the development data to select $\lambda$, and use the final test set once to assess the selected model. The model assessed is the particular fitted model trained on the development data. The test result is not automatically an estimate of the performance of a different model produced by changing the training data or fitting procedure.

Set aside the test data.
Make the split before using outcome information to choose transformations, predictors, or tuning settings. Keep test outcomes unavailable during development.

Tune using development data.
Use the designated resampling procedure to select the penalty and other model choices, without incorporating test outcomes.

Fix the selected model and metric.
Record the chosen specification and decide which performance measures will answer the prediction question.

Predict and evaluate.
Generate test predictions from the fixed model, calculate the prespecified measures, and report the test-set size and relevant limitations.

What Does a Test-Set Estimate Measure?

Let $\hat f$ denote a fitted prediction rule that is fixed before the test outcomes are examined. For test observation $j$, let $Y_j$ be the observed response and let $\hat f(X_j)$ be its prediction from predictors $X_j$. For a specified loss function $L$, the average test loss is

\widehat R_{\mathrm{test}} = \frac{1}{n_{\mathrm{test}}} \sum_{j=1}^{n_{\mathrm{test}}} L\!\left(Y_j,\hat f(X_j)\right).

This formula gives each test observation equal weight. It is appropriate when the test observations are independent draws from the target population, the fixed model is evaluated using the same prediction task intended for use, and the loss has a finite expectation. If the test sample has a different sampling design or does not represent the target population, this average may not estimate the intended performance without an appropriate adjustment.

The reason the average can estimate population performance is direct. For a fixed $\hat f$, suppose the test pairs $(X_j,Y_j)$ are independent and identically distributed as future target-population pairs, and the loss has finite expectation. Define the fixed model's population risk as $R(\hat f)=E[L(Y,\hat f(X))]$. Each test loss then has expectation $R(\hat f)$, so linearity of expectation gives

E\!\left[\widehat R_{\mathrm{test}}\mid \hat f\right] = \frac{1}{n_{\mathrm{test}}} \sum_{j=1}^{n_{\mathrm{test}}} E\!\left[L(Y_j,\hat f(X_j))\mid \hat f\right] = R(\hat f).

The conditioning is important: this justification treats the fitted model as fixed. If test results influence which model is chosen, the model is no longer fixed independently of those results, and the argument does not justify reporting the selected test performance as an independent assessment. Also, an unbiased estimate need not be precise: a small test set can yield a noisy result.

Worked Examples of Final Evaluation

Worked Example: Test RMSE for a Selected Penalized Regression Model

A hypothetical insurer uses a penalized regression model to predict claim cost in thousands of dollars. Cross-validation on the development data selected $\lambda=0.08$; the reported cross-validation RMSE was 2.00 thousand dollars. The model and its preprocessing are now fixed. On four untouched test records, observed costs and predictions, in thousands, are respectively $(12,10)$, $(8,9)$, $(15,12)$, and $(7,9)$. Calculate test RMSE and compare it with the cross-validation estimate. Assume equal weighting and that the test records represent the same prediction population.

Method. For squared-error loss, the test RMSE is the square root of the mean squared residual. The residual is observed cost minus predicted cost, in thousands of dollars.

The four residuals are $12-10=2$, $8-9=-1$, $15-12=3$, and $7-9=-2$. Their squared values are $2^2=4$, $(-1)^2=1$, $3^2=9$, and $(-2)^2=4$. The sum of squared residuals is $4+1+9+4=18$.

The mean squared error is $18/4=4.5$, in squared thousands of dollars. Therefore,

\mathrm{RMSE}_{\mathrm{test}} = \sqrt{\frac{18}{4}} = \sqrt{4.5} \approx 2.1213 \quad\text{thousand dollars}.

The test RMSE is $2.1213-2.00=0.1213$ thousand dollars higher than the cross-validation estimate. The measures are on the same scale, so this is a direct descriptive comparison, not proof that the model's population performance differs by exactly that amount. The test result corresponds to the fixed model and test sample described. Common mistake avoided: reporting the mean squared error, 4.5, as though it were RMSE. Taking the square root is necessary to return to the original outcome scale.

Worked Example: Test Log Loss for a Binary Outcome

A hypothetical maintenance team predicts whether a machine will fail within 30 days. A penalized logistic model has already been selected using development data. Its test predictions are $(y,p)=(1,0.80),(0,0.25),(1,0.60),(0,0.10)$, where $y$ is the observed outcome and $p$ is the predicted probability of failure. Cross-validation log loss was 0.2600. Calculate test log loss and compare the values. Use natural logarithms and equal weights.

Method. For a binary outcome $y\in\{0,1\}$ and a predicted probability $p$ strictly between 0 and 1, the log loss is $-[y\ln(p)+(1-y)\ln(1-p)]$. It penalizes a prediction according to the probability assigned to the outcome that actually occurred.

For the first record, $y=1$ and $p=0.80$, so the loss is $-\ln(0.80)=0.22314$. For the second, $y=0$ and $p=0.25$, so the loss is $-\ln(1-0.25)=-\ln(0.75)=0.28768$. For the third, $y=1$ and $p=0.60$, giving $-\ln(0.60)=0.51083$. For the fourth, $y=0$ and $p=0.10$, giving $-\ln(0.90)=0.10536$.

The total loss is $0.22314+0.28768+0.51083+0.10536=1.12701$. Dividing by four gives test log loss $1.12701/4=0.28175$, or 0.2818 to four decimal places. This is 0.0218 higher than the cross-validation estimate of 0.2600. Lower log loss is better, but this observed difference alone does not establish that future performance will be worse by 0.0218. Common mistake avoided: for a nonfailure, using $-\ln(p)$ rather than $-\ln(1-p)$. The loss must use the probability assigned to the outcome that occurred.

Worked Example: Why a Test-Driven Penalty Change Is Not a Final Assessment

A development procedure selected model A, a penalized regression fit at one value of $\lambda$. Before viewing test outcomes, the analyst specified test RMSE as the performance measure. On four test records, model A has squared errors $1,1,4,4$. After seeing those results, the analyst considers model B, which has squared errors $0,1,1,4$, and reports whichever model has the lower test RMSE. Calculate both values and decide what may be reported as the final assessment of model A.

Method. Calculate each model's mean squared error and take its square root. Since both models are evaluated on the same four records, the arithmetic is comparable; however, looking at test results to choose between them changes the role of the test set.

For model A, the sum of squared errors is $1+1+4+4=10$. Its mean squared error is $10/4=2.5$, so its test RMSE is $\sqrt{2.5}\approx1.5811$. For model B, the sum is $0+1+1+4=6$, the mean squared error is $6/4=1.5$, and the test RMSE is $\sqrt{1.5}\approx1.2247$.

Because model A was selected before test outcomes were examined, its prespecified test RMSE of 1.5811 is the relevant assessment of that selected model. Model B's lower value is a result observed after using the test set to consider a revision. It can be reported transparently as exploratory, but it cannot be presented as an independent final assessment of a model selected without test information. To assess a revised choice independently, the analyst needs new untouched assessment data or a development procedure that does not reuse this test set as if it were untouched. Common mistake avoided: choosing model B because it performed better on the test set and then reporting its 1.2247 as though the test set had not participated in model selection.

Comparing Cross-Validation and Test Results

Cross-validation and final test performance answer related but not necessarily identical questions. A cross-validation estimate summarizes performance across held-out folds under the resampling and fitting procedure used during tuning. The test estimate summarizes the fixed selected model's performance on the particular untouched sample. Differences can arise from sampling variability, different training-set sizes within the resampling procedure, and chance variation in the observations held out. The comparison is most interpretable when both estimates use the same outcome scale, loss definition, weighting convention, and target population.

It is useful to report the cross-validation estimate as a tuning result and the test estimate as a separate final assessment. If cross-validation results include an estimate of their variability, that uncertainty should not be silently treated as a confidence interval for the test result. Likewise, one observed test value does not describe how much that value might vary across new test samples. Report the sample size and avoid strong conclusions based on a small numerical difference.

An evaluation plan should also make clear which measures are primary. Computing several prespecified measures on the same test set can be informative; repeatedly examining results and then choosing the most favorable metric, subgroup, threshold, or model is a form of test-set reuse. A metric chosen after seeing outcomes should not be portrayed as though it had been specified in advance.

Preventing Leakage and Preserving the Test Set

A test set is not untouched merely because it was excluded from the final model-fitting call. Any data-dependent decision that uses its outcomes can leak information into the development process. For example, selecting predictors after inspecting their test associations, choosing an imputation rule based on test outcomes, or adjusting a probability threshold to improve test accuracy all compromise the final assessment. Data transformations and encodings should be learned from the permitted development data and then applied consistently to test predictors. Test outcomes must not guide those transformations.

Keep an auditable record of the split, resampling procedure, selected $\lambda$, prediction scale, test metrics, and any decisions made after evaluation. If a test result prompts a material revision, label the result as exploratory and do not reuse it as an independent confirmation of the revised model. This discipline matters because repeated attempts can make a chance-favorable result look like genuine predictive improvement.

Key takeaway: Tune the penalty using development data, fix the model and performance measures, and then evaluate once on untouched test observations. Compare the test result with tuning estimates cautiously; if test outcomes change the model or evaluation choices, the test set no longer provides an independent final assessment.

Check Your Understanding

For each problem, distinguish a final test assessment from a development decision and show the calculation where requested.

A fixed penalized model predicts dollar claim amounts of 20, 30, and 40 for three test records whose observed amounts are 21, 28, and 42. Calculate test RMSE in dollars. (Answer: residuals are $-1,2,-2$; MSE is $9/3=3$; RMSE is $\sqrt{3}\approx\$1.7321$.)
A test set has outcomes and predicted probabilities $(1,0.70),(0,0.20),(1,0.90)$. Calculate average log loss using natural logarithms, to four decimal places. (Answer: $[-\ln(0.70)-\ln(0.80)-\ln(0.90)]/3\approx0.2284$.)
Cross-validation selected a penalty whose test residuals are $1,-1,2,0,-2$, in thousands of dollars. Calculate the test RMSE and its difference from a cross-validation RMSE of 1.30 thousand dollars. (Answer: squared residuals sum to 10; test RMSE is $\sqrt{10/5}=\sqrt{2}\approx1.4142$ thousand dollars, which is 0.1142 thousand dollars higher.)
An analyst scales predictors using means and standard deviations calculated from the complete data set before splitting into development and test sets. The test outcomes are not used. Is the test set fully untouched, and what is the correction? (Answer: No. The test predictors influenced preprocessing. Estimate scaling parameters from development data only, then apply those fixed parameters to the test predictors.)
Model A was selected using development data. On four test records, its squared errors are $1,1,4,4$; a model suggested after viewing test results has squared errors $0,1,1,4$. Calculate both RMSE values and state which can serve as the independent final assessment of the preselected model. (Answer: model A has RMSE $\sqrt{10/4}\approx1.5811$; model B has RMSE $\sqrt{6/4}\approx1.2247$. Only model A's prespecified result is the independent assessment of the model selected before test inspection.)

What You'll Learn