Why Prediction-Error Uncertainty Has Two Sources
A reported prediction-error estimate is not automatically a single, unambiguous measure of uncertainty. If a model is evaluated on a finite test sample, its error estimate varies with which evaluation cases happen to be observed. If the model is refitted on a different training sample, its predictions—and therefore its prediction error—may also vary. A bootstrap analysis must identify which source of variation it is intended to measure.
This distinction matters when using a test set to assess a predictive model. Resampling test cases while keeping the fitted model fixed answers a different question from resampling training cases and refitting the model. Neither bootstrap distribution should be described simply as “the model’s uncertainty” without stating what was resampled.
The metric must be fixed before generating bootstrap replicates. For example, for squared-error loss the test mean squared error is the average of the squared differences between observed outcomes and predictions. For classification error, it is the proportion of cases whose predicted class is incorrect. If the chosen metric is root mean squared error, compute the square root separately for each bootstrap replicate of mean squared error; do not average squared errors and take a single square root afterward if the target metric is the average of replicate RMSE values.
| Resampled cases | What is refitted? | Main uncertainty represented |
|---|---|---|
| Evaluation cases; keep the fitted predictor fixed | Nothing | Variation from having a finite evaluation sample, conditional on the fitted model |
| Training cases; refit the complete procedure | The predictor is refitted for every replicate | Variation in fitted predictions and their evaluated performance from the training sample |
| Training and evaluation cases, resampled separately | Refit for each training replicate | Variation from both samples, subject to the chosen evaluation design |
Bootstrap the Evaluation Sample With the Model Fixed
Suppose an evaluation sample contains \(n\) independent, representative cases that are separate from the training data. First calculate the prediction loss for each case using the already-fitted predictor. For a bootstrap replicate, sample \(n\) complete evaluation cases with replacement, and calculate the same metric on that resample. Resampling complete cases preserves the observed pairing between each predictor vector and its outcome.
For a mean loss, the observed test error is the average of the \(n\) case losses. The bootstrap replicates approximate how that average could vary if a different evaluation sample of the same size were drawn from the target population. This is uncertainty about estimating the fixed model’s risk—not uncertainty about what model would have been fitted using a different training sample.
Worked Example: Bootstrap Interval for a Fixed Model’s Test MSE
A hypothetical fitted model is evaluated on four independent test cases. Its squared errors, in squared response units, are \(0, 2, 4,\) and \(6\). Five evaluation bootstrap samples, each of size four, produce these loss values: (a) \(0,0,2,4\); (b) \(2,2,4,6\); (c) \(0,4,4,6\); (d) \(0,2,6,6\); and (e) \(2,4,6,6\). Calculate the observed test MSE and an illustrative 80% percentile interval. The model is held fixed in all five replicates.
Method. Average the observed squared errors, then average the four losses in each bootstrap sample. For this illustration, define an empirical percentile as the ordered replicate at rank \(\lceil Bp\rceil\), where \(B=5\), \(p\) is the percentile proportion, and \(\lceil\cdot\rceil\) rounds up to the next integer.
The observed test MSE is $$ \frac{0+2+4+6}{4}=\frac{12}{4}=3. $$
The five bootstrap MSEs are $$ \begin{aligned} \text{(a)}\quad&\frac{0+0+2+4}{4}=\frac{6}{4}=1.5,\\ \text{(b)}\quad&\frac{2+2+4+6}{4}=\frac{14}{4}=3.5,\\ \text{(c)}\quad&\frac{0+4+4+6}{4}=\frac{14}{4}=3.5,\\ \text{(d)}\quad&\frac{0+2+6+6}{4}=\frac{14}{4}=3.5,\\ \text{(e)}\quad&\frac{2+4+6+6}{4}=\frac{18}{4}=4.5. \end{aligned} $$
In increasing order, the replicate values are \(1.5,3.5,3.5,3.5,4.5\). The 10th-percentile rank is \(\lceil5(0.10)\rceil=1\), and the 90th-percentile rank is \(\lceil5(0.90)\rceil=5\). Thus the illustrative 80% percentile interval is \([1.5,4.5]\) squared response units.
Check. The interval is centered at neither a required value nor an exact population risk: bootstrap intervals need not be symmetric around the observed MSE. With only five replicates, the endpoints are extremely coarse; this calculation illustrates the method, not a reliable final interval. Common mistake avoided: refitting the model here would mix training-sample variation into an interval intended to describe evaluation-sample variation.
Bootstrap the Training Sample and Refit
To study training-sample variation, resample the training cases with replacement and refit the entire specified modeling procedure in each replicate. “Entire procedure” includes any fitting steps that are part of producing the predictor. Evaluate each refitted predictor on a separate evaluation sample or another evaluation design that does not reuse that replicate’s bootstrap training cases as though they were new cases.
If one fixed evaluation sample is used for all the refitted predictors, the resulting spread reflects how the training resamples change performance on that particular evaluation sample. It does not include the additional variation that would arise from drawing a new evaluation sample. In addition, training bootstrap replicates are not automatically a substitute for an independent test sample: evaluating a fitted model on the same cases used to fit it can give an optimistic error estimate.
Worked Example: Training-Sample Variation for a Mean Predictor
A hypothetical model predicts the same value for every case: the mean of the training outcomes. The original training outcomes are \(2,4,6\). Five bootstrap training samples are \((2,2,4)\), \((2,4,6)\), \((4,6,6)\), \((2,2,6)\), and \((2,6,6)\). For each, refit the mean predictor and calculate its MSE on a separate, fixed evaluation set with outcomes \(3,5\).
Method. Calculate each bootstrap training mean, use that mean as the prediction for both evaluation outcomes, and average the two squared errors.
The five fitted predictions are $$ \frac{2+2+4}{3}=\frac{8}{3},\quad \frac{2+4+6}{3}=4,\quad \frac{4+6+6}{3}=\frac{16}{3},\quad \frac{2+2+6}{3}=\frac{10}{3},\quad \frac{2+6+6}{3}=\frac{14}{3}. $$
For prediction \(8/3\), the evaluation errors are \(3-8/3=1/3\) and \(5-8/3=7/3\), so its MSE is $$ \frac{(1/3)^2+(7/3)^2}{2}=\frac{1+49}{18}=\frac{25}{9}. $$
For prediction \(4\), the errors are \(-1\) and \(1\), giving MSE \((1+1)/2=1\). For prediction \(16/3\), the errors are \(-7/3\) and \(-1/3\), giving MSE \((49/9+1/9)/2=25/9\). For prediction \(10/3\), the errors are \(-1/3\) and \(5/3\), giving MSE \((1/9+25/9)/2=13/9\). For prediction \(14/3\), the errors are \(-5/3\) and \(1/3\), giving MSE \((25/9+1/9)/2=13/9\).
The five training-bootstrap MSEs are therefore \(25/9,1,25/9,13/9,13/9\). Their mean is $$ \frac{25/9+9/9+25/9+13/9+13/9}{5} =\frac{85}{45}=\frac{17}{9}\approx1.8889. $$
Check. The original training mean is \(4\), and its MSE on the fixed evaluation outcomes is \(\{(3-4)^2+(5-4)^2\}/2=1\), matching the replicate that resampled all three training outcomes once. The other replicates differ because the fitted prediction changes. Common mistake avoided: treating the five replicate errors as evaluation-only variation would ignore that each replicate has a different fitted model.
Combining the Two Sources of Variation
A nested bootstrap can represent both sources: draw a training bootstrap sample and refit, then independently resample evaluation cases and calculate the metric for that fitted model. Repeating this pair of steps produces a joint distribution of prediction-error estimates. Independence between training and evaluation samples is important to this interpretation; resampling evaluation cases does not make a training set independent if the original design linked the two.
The variance decomposition gives a concise justification for separating the sources. Write \(T\) for the error estimate produced by the resampling scheme and \(D_{\mathrm{tr}}\) for a training sample. Conditional on \(D_{\mathrm{tr}}\), the model is fixed, while evaluation resampling can still vary \(T\). The law of total variance states:
The first term measures variation among conditional average errors across training samples. The second is the average evaluation-sample variance remaining after a training sample is fixed. This identity follows by writing \(T-E[T]=\{T-E[T\mid D_{\mathrm{tr}}]\}+\{E[T\mid D_{\mathrm{tr}}]-E[T]\}\). The two bracketed terms have covariance zero because the conditional expectation of the first term given \(D_{\mathrm{tr}}\) is zero. Taking variances therefore gives the displayed sum.
Worked Example: Decomposing a Nested Bootstrap Variance
Suppose a small illustrative nested bootstrap has two equally weighted training replicates. For the first training replicate, two evaluation-bootstrap error estimates are \(2\) and \(4\); for the second, they are \(4\) and \(8\). Use population-variance divisors for this finite illustrative calculation and decompose the variance of all four estimates.
Method. First calculate the mean and variance of evaluation estimates within each training replicate. Then calculate the variance of the two conditional means and average the two within-training variances.
For the first training replicate, the conditional mean is \((2+4)/2=3\), and its variance is \(\{(2-3)^2+(4-3)^2\}/2=1\). For the second, the mean is \((4+8)/2=6\), and its variance is \(\{(4-6)^2+(8-6)^2\}/2=4\).
The overall mean of the conditional means is \((3+6)/2=4.5\). Their variance is \(\{(3-4.5)^2+(6-4.5)^2\}/2=2.25\). The average within-training variance is \((1+4)/2=2.5\). Thus the decomposed total variance is \(2.25+2.5=4.75\).
Directly, the four values \(2,4,4,8\) have mean \(4.5\), and variance \(\{6.25+0.25+0.25+12.25\}/4=19/4=4.75\), confirming the decomposition. These few replicates are only an arithmetic illustration; a practical analysis needs enough replicates to estimate its distribution with suitable Monte Carlo precision. Common mistake avoided: adding standard deviations or interpreting the within-training component as the total uncertainty would not perform this variance decomposition.
Intervals, Assumptions, and Common Traps
A percentile interval uses chosen quantiles of the bootstrap replicate metric values as its endpoints. Its interpretation depends on the resampling scheme. An interval from test-case resampling is conditional on the fitted model and describes uncertainty in the evaluation estimate under the assumption that observed evaluation cases represent independent draws from the target population. It does not, by itself, describe variation from collecting a new training sample.
The usual case bootstrap assumes the resampled units are exchangeable observations from the population relevant to the target prediction task. It can be unreliable when the evaluation sample is very small, when important types of cases are absent, or when the evaluation data do not represent the intended prediction population. The bootstrap does not repair biased sampling, distribution shift, or a metric chosen after inspecting results. The prediction-error metric and the intended target population must be stated clearly.
A particularly common exam trap is to take a bootstrap sample of training cases, score the refitted model on those same sampled cases, and call the resulting average an estimate of out-of-sample prediction error. The model has been fitted using those cases, so this calculation is not an independent evaluation. Another trap is to resample only the observed loss values when the actual target is a procedure that includes refitting: that conditions on the original fitted model and omits training variation.
Check Your Understanding
For each problem, identify the relevant resampling calculation and check the arithmetic before interpreting the result.
- A fixed predictor has losses \(1,3,5\) on three evaluation cases. What is the observed mean loss, and what is the mean loss for the bootstrap replicate \((1,1,5)\)? (Answer: observed mean \(3\); replicate mean \(7/3\).)
- A fixed classifier makes 2 errors on an evaluation sample of 10 cases. A bootstrap replicate contains 8 resampled cases, of which 3 are errors. Give the observed error rate and that replicate’s error rate. (Answer: \(0.2000\) and \(0.3750\).)
- A mean predictor is refitted from the training bootstrap sample \((1,1,5)\). It is evaluated on two fixed outcomes, \(2\) and \(4\). Find its prediction and evaluation MSE. (Answer: prediction \(7/3\); MSE \(13/9\).)
- In a nested bootstrap, two equally weighted training replicates have conditional means \(2\) and \(6\), and conditional evaluation variances \(1\) and \(9\). Find the between-training variance, average within-training variance, and total variance. (Answer: \(4\), \(5\), and \(9\), respectively.)
- A percentile interval is formed by resampling evaluation cases while leaving the fitted predictor unchanged. Does this interval include uncertainty from drawing a new training sample? (Answer: No; it describes evaluation-sample uncertainty conditional on the fitted predictor.)