Single Holdout Estimates of Predictive Performance

Why Hold Out Test Data?

A model can fit its training data well without predicting new observations equally well. Measuring performance on the same records used to fit a model generally gives an overly favorable view of how it will perform on future data. A single holdout estimate addresses this by reserving part of the available data, fitting the model on the remainder, and evaluating the fitted model on records it did not use.

The resulting test error is useful, but it is not a universal property of a model-building method. It is an estimate based on one fitted model and one particular set of test observations. Its interpretation depends on how the split was made, whether the test observations resemble the intended future population, and how many test observations are available.

Definition: A single holdout split partitions a data set into a training set and a test set. The model is fitted using the training set only. Its test error is the average of a specified loss over the test observations. The test set must remain separate from model fitting and decisions that would otherwise use test performance.

Estimating Error on One Test Set

Suppose the training data are denoted by $\mathcal{D}_{\text{tr}}$, the test data contain $n_{\text{te}}$ observations, and fitting the chosen procedure to the training data produces the predictor $\hat f_{\mathcal{D}_{\text{tr}}}$. A loss function $L$ assigns a numerical penalty to a prediction and its observed outcome. The single holdout estimate is

\widehat{R}_{\text{te}} = \frac{1}{n_{\text{te}}} \sum_{i\in\mathcal{D}_{\text{te}}} L\!\left(Y_i,\hat f_{\mathcal{D}_{\text{tr}}}(X_i)\right).

Here $X_i$ denotes the predictors for test observation $i$, and $Y_i$ is its observed outcome. The particular loss must match the question being asked. For classification, the zero-one loss is 1 when the predicted class is wrong and 0 when it is correct; its average is the misclassification rate. For regression, a common loss is squared error, $(Y_i-\hat f(X_i))^2$, although absolute error is another option. Different losses summarize different aspects of performance.

If misclassification rate is $e$, accuracy is $1-e$, provided every test observation receives exactly one predicted class. Accuracy and error therefore contain the same information in this setting, but the error rate makes the fraction of incorrect predictions explicit. In an imbalanced classification problem, overall accuracy can hide poor performance on a less common class, so class-specific rates may also be needed.

Interpretation: For a fixed fitted predictor, its prediction risk is the expected loss on a new observation from the target population, conditional on the training data. A test-set average estimates that risk when the test observations are independent of the training data and are representative, independent draws from that target population. It does not by itself estimate how performance varies across different training samples.

Why the Test Average Estimates Risk

The justification is a sample-average argument. Treat the training data and hence the fitted predictor as fixed. If test observations are independent draws from the target population, then each test loss has the same conditional expectation: the prediction risk of that fitted predictor. Averaging those losses preserves the expectation.

E\!\left[\widehat{R}_{\text{te}}\mid\mathcal{D}_{\text{tr}}\right] = \frac{1}{n_{\text{te}}} \sum_{i\in\mathcal{D}_{\text{te}}} E\!\left[ L\!\left(Y_i,\hat f_{\mathcal{D}_{\text{tr}}}(X_i)\right) \mid\mathcal{D}_{\text{tr}} \right] = R\!\left(\hat f_{\mathcal{D}_{\text{tr}}}\right).

The equality follows because each conditional expected test loss is the same risk, so the average of $n_{\text{te}}$ identical expectations is that risk. This argument does not say the observed test error must equal the risk: different test samples produce different averages. Nor does it establish unbiasedness if the test observations are selected in a way that changes their distribution, if they are dependent in a way the calculation ignores, or if their outcomes have influenced the fitted model.

There are two sources of variability to keep distinct. Even for one fixed fitted predictor, a different test sample could produce a different test error. If the training sample and model fit also changed, the predictor and its risk could change as well. A single split does not measure that second source of variation. Its estimate is conditional on the particular training set and fitted model.

Worked Examples

Worked Example: Test Error and Its Approximate Sampling Uncertainty

A hypothetical insurer has 1,000 independent claim records drawn from the population to which it intends to apply a fraud-screening classifier. It randomly assigns 800 records to training and 200 to testing. The classifier, fitted using only the 800 training records, makes 12 false-negative and 19 false-positive predictions on the test set. Calculate the test error, accuracy, and an approximate standard error for the test error.

Method. Use the test-set fraction misclassified for error. For an approximate standard error, treat each test error indicator as a Bernoulli observation with a common error probability, conditional on the fitted classifier.

Step 1: Count the test errors. There are $12+19=31$ misclassified records among 200 test records.

\widehat{R}_{\text{te}}=\frac{31}{200}=0.155.

The estimated test error is 0.155, or 15.5%.

Step 2: Calculate accuracy. The number correctly classified is $200-31=169$. Thus accuracy is $169/200=0.845$, or 84.5%. As a check, $1-0.155=0.845$.

Step 3: Estimate the standard error. Under the stated Bernoulli approximation, substitute the observed error rate for the unknown error probability:

\widehat{\operatorname{SE}}(\widehat{R}_{\text{te}}) = \sqrt{\frac{0.155(1-0.155)}{200}} = \sqrt{\frac{0.130975}{200}} \approx 0.0256.

The estimated standard error is about 2.56 percentage points. A rough normal-approximation interval is $0.155\pm1.96(0.0256)$, or approximately 0.1048 to 0.2052. This interval is only an approximation; it describes test-sampling uncertainty under the assumptions, not uncertainty across refitted training samples.

Common mistake avoided. Reporting 15.5% as an exact, fixed future error rate ignores sampling variation. The value is an estimate from this test set. Also, the false negatives and false positives should both be counted as errors for the overall misclassification rate.

Worked Example: Squared-Error and Absolute-Error Summaries

A hypothetical model predicts the next-month aggregate claim cost for five independent test groups. Both observed and predicted costs are recorded in thousands of dollars. The observed-minus-predicted residuals, in the same order as the five groups, are $-1, 2, -2, 3,$ and $0$. Calculate the test mean squared error, root mean squared error, and mean absolute error.

Method. Square each residual for squared-error loss, take the mean, and then take its square root for root mean squared error. For absolute-error loss, take absolute values before averaging.

Step 1: Calculate squared residuals. The five squared residuals are $(-1)^2=1$, $2^2=4$, $(-2)^2=4$, $3^2=9$, and $0^2=0$. Their sum is $1+4+4+9+0=18$, in squared thousands of dollars.

Step 2: Calculate mean squared error and root mean squared error.

\operatorname{MSE}_{\text{te}}=\frac{18}{5}=3.6 \quad\text{(thousand dollars)}^2, \qquad \operatorname{RMSE}_{\text{te}}=\sqrt{3.6}\approx1.89737 \quad\text{thousand dollars}.

Thus the RMSE is approximately $1.89737\times \$1{,}000=\$1{,}897.37$, rounded to the nearest cent.

Step 3: Calculate mean absolute error. The absolute residuals are $1,2,2,3,0$. Their sum is 8, so the mean absolute error is $8/5=1.6$ thousand dollars, or $1,600.

Check. The RMSE is larger than the mean absolute error here, as is expected for these residuals because squaring gives relatively more weight to the largest residual. The two summaries are in dollars after converting units, but they are not interchangeable: they apply different penalties.

Common mistake avoided. The mean squared error is in squared units, not thousands of dollars. Taking its square root returns the measure to the original cost units; omitting that step when reporting an RMSE would give the wrong units and quantity.

Worked Example: Class Imbalance and the Meaning of Overall Error

A hypothetical classifier is tested on 100 policy applications: 20 are actually high-risk and 80 are actually standard-risk. It correctly identifies 15 of the 20 high-risk applications and correctly identifies 72 of the 80 standard-risk applications. Calculate overall test error, sensitivity, specificity, and balanced accuracy.

Method. Count the errors over all test records for overall error. Calculate sensitivity and specificity within their respective actual classes. Balanced accuracy is the average of sensitivity and specificity.

Step 1: Find the errors. There are $20-15=5$ false negatives and $80-72=8$ false positives. The total number of errors is $5+8=13$, so the overall test error is $13/100=0.13$, or 13%. Accuracy is $1-0.13=0.87$, or 87%.

Step 2: Calculate class-specific rates. Sensitivity is the fraction of actual high-risk applications correctly identified: $15/20=0.75$. Specificity is the fraction of actual standard-risk applications correctly identified: $72/80=0.90$.

Step 3: Calculate balanced accuracy.

\text{Balanced accuracy} = \frac{0.75+0.90}{2} = 0.825.

Balanced accuracy is 82.5%. It weights the two class-specific rates equally; overall accuracy instead weights performance according to the observed class proportions.

Common mistake avoided. Averaging the two class error rates, $(0.25+0.10)/2=0.175$, does not give overall test error; it gives the average class error rate. For overall error, weight each class error rate by its share of the test set:

\frac{20}{100}(0.25)+\frac{80}{100}(0.10) = 0.05+0.08 = 0.13.

The weighted result agrees with the direct count of 13 errors out of 100. Both overall error and balanced accuracy can be informative, but they answer different questions.

What a Single Split Can and Cannot Tell You

A single holdout estimate is most defensible when the split reflects the intended prediction task: training observations are available before prediction, and test observations are representative of the future cases of interest. Randomly assigning independent, identically distributed observations to the two sets is one common design. If the future population differs from the test set, the test estimate may not describe future performance even when the arithmetic is correct.

Test size affects precision. For a classification error indicator with conditional error probability $p$, independent test observations have an error-rate variance of $p(1-p)/n_{\text{te}}$. Replacing the unknown $p$ with the observed test error gives the familiar approximate standard error used in the first example. A smaller test set generally produces a less precise estimate. For regression losses, the standard error of their mean depends on the variability of the individual losses; a few unusually large losses can make that variability substantial.

The test estimate can also vary because a different random partition would place different observations in the test set and would produce a different training set. The one-split result does not reveal how large that partition-to-partition variation is. Its numerical precision should not be confused with certainty about performance under other training samples, future populations, or modeling decisions.

Keep the test set out of the process used to choose or adjust a model. If repeated inspection of test performance guides those decisions, the test set has indirectly become part of the fitting process; the reported performance can then be optimistic. The basic safeguard is to settle the model-building decisions without using test outcomes, and use the test set for a final evaluation.

Key takeaway: A single test error is an average loss on one held-out sample for one fitted predictor. Interpret it alongside the loss definition, test-set size, class or outcome composition, and assumptions connecting the test observations to future data.

Check Your Understanding

For each problem, identify the requested test-set quantity and keep its interpretation tied to the stated test data.

A classifier makes 18 mistakes on 150 independent test records. What are its test error and accuracy? (Answer: error $=0.1200$; accuracy $=0.8800$.)
On four regression test observations, the residuals in dollars are $-2, 1, 0,$ and $3$. Calculate the test MSE and test MAE. (Answer: MSE $=3.5$ dollars squared; MAE $=1.5$ dollars.)
A test set contains 30 positive and 70 negative cases. The classifier misses 6 positive cases and incorrectly labels 7 negative cases as positive. Calculate overall error and balanced accuracy. (Answer: overall error $=13/100=0.1300$; balanced accuracy $=[(24/30)+(63/70)]/2=0.8500$.)
A fixed classifier has observed test error $0.20$ on 100 independent test observations. Using the plug-in Bernoulli formula, calculate the approximate standard error. (Answer: $\sqrt{0.20(0.80)/100}=0.0400$.)
Suppose the test error estimate is small, but the test records were selected from a population with a different outcome mix than the future cases. Is the estimate necessarily representative of future error? (Answer: No. The test average estimates performance for the test distribution; a changed target distribution can make it unrepresentative.)

What You'll Learn