Why Compare Models on the Same Cases?
Suppose two predictive methods are evaluated on the same held-out cases. A direct comparison should account for the fact that a case that is difficult for one method may also be difficult for the other. The natural unit of comparison is therefore not either model’s loss by itself, but the difference between the two losses on each case.
This pairing matters when estimating uncertainty as well as when calculating the observed comparison. If the pairing is ignored, the resampling procedure acts as though one model’s loss on a case can be matched with the other model’s loss on an unrelated case. That is a different comparison and generally does not preserve the information in the evaluation data.
The difference should be defined in a consistent direction and kept in that direction throughout the analysis. For example, with \(D_i=L_{Ai}-L_{Bi}\), positive values mean A had more loss on case \(i\). If the metric is an error indicator, \(L_{Ai}=1\) means A made an error on that case and \(L_{Ai}=0\) means it did not. If the metric is squared error, \(L_{Ai}=(y_i-\hat{y}_{Ai})^2\), where \(y_i\) is the observed response and \(\hat{y}_{Ai}\) is A’s prediction.
Paired Bootstrap Resampling
For a paired bootstrap comparison, treat the case-level pair \((L_{Ai},L_{Bi})\), or equivalently the paired difference \(D_i\), as one resampling unit. Draw \(n\) cases with replacement from the \(n\) held-out cases. For every selected case, retain both methods’ losses together. The mean difference in bootstrap replicate \(b\) is \(\bar{D}^{*(b)}\).
Here \(I_j^{(b)}\) is the index of the held-out case selected for draw \(j\) in replicate \(b\). Repeated indices are allowed because sampling is with replacement. The resulting \(B\) replicate means form an empirical bootstrap distribution for the average paired difference. Percentiles of that distribution can be used to construct a bootstrap interval.
For each held-out case, compute both methods’ losses using the same observed outcome and the same performance measure.
Choose a direction, such as \(D_i=L_{Ai}-L_{Bi}\), and calculate \(\bar{D}\).
For each bootstrap replicate, sample \(n\) indices with replacement and apply each selected index to both methods’ losses.
Calculate the replicate mean differences and use their spread or percentiles to describe uncertainty in the comparison.
The bootstrap has a clear target here: uncertainty in the average loss difference for a population of cases represented by the held-out sample, conditional on the predictions being evaluated. It does not, by itself, include variability from drawing new training data and refitting either method. To study that additional variability, the model-fitting process would also need to be repeated within an appropriate resampling design.
The ordinary case bootstrap assumes that the evaluation cases are independent and reasonably representative of the target population. If observations are dependent or naturally grouped, the resampling unit must respect that structure—for example, resampling clusters rather than individual records when records within a cluster are dependent. The paired comparison still requires both methods’ results to stay together for each selected unit.
Why Keeping the Pairing Matters
The benefit of pairing can be seen from the variance of a difference. If \(L_A\) and \(L_B\) denote the losses from the two methods on a randomly selected case, then
This identity follows by expanding the variance of a difference: \(\operatorname{Var}(L_A-L_B)=\operatorname{Var}(L_A)+\operatorname{Var}(-L_B)+2\operatorname{Cov}(L_A,-L_B)\). Since \(\operatorname{Var}(-L_B)=\operatorname{Var}(L_B)\) and \(\operatorname{Cov}(L_A,-L_B)=-\operatorname{Cov}(L_A,L_B)\), the stated expression follows. When the methods tend to have high or low losses on the same cases, their losses are positively associated. The covariance term then reduces the variance of their paired difference. Resampling each model’s losses separately discards this covariance.
Worked Example: Comparing Squared Errors on the Same Cases
Two methods predict hypothetical claim amounts for five held-out policies. The observed amounts are used to calculate squared-error losses. The losses, in squared thousands of dollars, are shown below. Compare the methods using \(D_i=L_{Ai}-L_{Bi}\), then calculate two paired bootstrap replicate means.
| Case | Method A loss | Method B loss | \(D_i=L_{Ai}-L_{Bi}\) |
|---|---|---|---|
| 1 | 9 | 4 | 5 |
| 2 | 4 | 9 | -5 |
| 3 | 16 | 9 | 7 |
| 4 | 1 | 4 | -3 |
| 5 | 9 | 1 | 8 |
Method and observed comparison. Subtract B’s loss from A’s loss case by case. The differences sum to \(5-5+7-3+8=12\), so \(\bar{D}=12/5=2.4\). As a check using the separate means, A’s mean loss is \((9+4+16+1+9)/5=39/5=7.8\), and B’s is \((4+9+9+4+1)/5=27/5=5.4\). Their difference is \(7.8-5.4=2.4\), as it must be. Thus B has lower average squared error on these five cases.
First paired bootstrap replicate. Suppose the sampled case indices are \(1,2,2,5,4\). The sampled differences are \(5,-5,-5,8,-3\). Their sum is \(5-5-5+8-3=0\), so \(\bar{D}^{*(1)}=0/5=0\).
Second paired bootstrap replicate. Suppose the sampled indices are \(3,3,1,5,5\). The sampled differences are \(7,7,5,8,8\). Their sum is \(7+7+5+8+8=35\), so \(\bar{D}^{*(2)}=35/5=7\).
Each replicate selected complete case-level comparisons, including repeated cases where the same index appeared more than once. The two replicate means do not replace the observed estimate \(2.4\); they illustrate how the estimate can vary under resampling. The common mistake avoided is to sample A’s and B’s losses using separate index draws, which would break the original case matching.
Paired Comparisons for Classification Loss
The same approach applies when the loss is a classification error indicator. On each case, the paired difference is \(1\) if only A makes an error, \(-1\) if only B makes an error, and \(0\) if both methods are either correct or incorrect. Consequently, only the cases on which the methods disagree about correctness contribute to the difference in their error rates. Cases on which both are correct or both are wrong remain part of the resampling data, but contribute zero to the mean difference.
Worked Example: Equal Error Rates but Different Case Outcomes
Two classifiers are evaluated on eight held-out records from a hypothetical screening exercise. A loss of 1 indicates a misclassification and 0 indicates a correct classification. Their losses are:
| Case | A loss | B loss | \(D_i\) |
|---|---|---|---|
| 1 | 1 | 0 | 1 |
| 2 | 1 | 1 | 0 |
| 3 | 0 | 1 | -1 |
| 4 | 0 | 0 | 0 |
| 5 | 1 | 0 | 1 |
| 6 | 0 | 0 | 0 |
| 7 | 0 | 1 | -1 |
| 8 | 0 | 0 | 0 |
Observed comparison. A makes \(1+1+0+0+1+0+0+0=3\) errors, giving an error rate of \(3/8=0.375\). B also makes \(0+1+1+0+0+0+1+0=3\) errors, also \(3/8=0.375\). The paired differences sum to \(1+0-1+0+1+0-1+0=0\), so \(\bar{D}=0/8=0\). The observed error rates are equal, although the methods do not make errors on exactly the same cases.
Paired bootstrap replicates. For indices \(1,2,3,5,6,6,7,8\), the selected differences are \(1,0,-1,1,0,0,-1,0\). Their sum is 0, giving a replicate mean of \(0/8=0\). For indices \(1,1,1,4,5,5,5,8\), the selected differences are \(1,1,1,0,1,1,1,0\). Their sum is 6, giving a replicate mean of \(6/8=0.75\).
These replicates show why equal observed error rates do not imply that every resampled comparison will be zero. The bootstrap repeatedly changes how often each case appears, while retaining the paired outcomes for each selected case. The common mistake avoided is to conclude that the methods behave identically just because their marginal error counts agree.
Using Percentiles to Describe Uncertainty
A percentile interval summarizes the central part of the bootstrap distribution. With \(B\) replicate means, sort them from smallest to largest. For a reproducible calculation in the next example, use the interpolation rule \(h=1+(B-1)p\) for percentile \(p\), with linear interpolation between the adjacent ordered values. A central 90% percentile interval uses the 5th and 95th percentiles. The chosen interval level and quantile convention should be stated when reporting results.
Worked Example: Calculating a Percentile Interval
A paired bootstrap produces \(B=20\) replicate mean loss differences for two hypothetical forecasting methods. The values, already sorted from smallest to largest, are:
\(-3,-2,-1,0,0,1,1,2,2,2,3,3,4,4,5,5,6,7,8,10\).
The observed mean paired difference is \(\bar{D}=3.0\), using the convention \(D_i=L_{Ai}-L_{Bi}\). Calculate the 90% percentile interval and interpret it.
Lower endpoint. For \(p=0.05\), \(h=1+(20-1)(0.05)=1.95\). This is 0.95 of the way from the first ordered value, \(-3\), to the second, \(-2\). The endpoint is \(-3+0.95[-2-(-3)]=-3+0.95=-2.05\).
Upper endpoint. For \(p=0.95\), \(h=1+(20-1)(0.95)=19.05\). This is 0.05 of the way from the nineteenth value, \(8\), to the twentieth, \(10\). The endpoint is \(8+0.05(10-8)=8.10\).
The resulting 90% percentile interval is \([-2.05,8.10]\). It includes zero, so this interval does not separate the average paired difference from zero at its stated level. The observed difference is positive, favoring B on the held-out sample, but the interval reflects substantial uncertainty about the average difference. Twenty replicates are used here only to make the arithmetic visible; a practical analysis would use many more replicates to stabilize estimated percentiles. The common mistake avoided is treating the observed difference as proof of a population advantage while ignoring the resampling uncertainty.
Common Mistakes and Interpretation
- Resampling the methods independently. Draw case indices once per replicate and apply them to both methods. Separate draws destroy the within-case pairing and do not estimate the same comparison.
- Reversing the sign. With \(D_i=L_{Ai}-L_{Bi}\), positive values favor B when lower loss is better. State the subtraction order before interpreting the result.
- Comparing different cases. A paired comparison requires both methods to be evaluated on the same held-out cases using the same loss definition. Otherwise, differences can reflect changes in case composition as well as changes in model performance.
- Overstating what the interval covers. Resampling fixed held-out predictions describes uncertainty associated with sampling evaluation cases. It does not automatically capture model-fitting variability, leakage, or a change in the target population.
- Ignoring dependence. If cases are clustered or time-dependent, ordinary independent case resampling may be unsuitable. The resampling unit must reflect the dependence structure while keeping each selected unit’s model comparisons paired.
Check Your Understanding
For each problem, retain the stated subtraction order and show the intermediate arithmetic before checking the answer.
- Four held-out cases have losses \((L_A,L_B)\) equal to \((5,7),(8,4),(3,3),(9,5)\). Calculate each \(D_i=L_{Ai}-L_{Bi}\), the observed mean difference, and which method has lower average loss. (Answer: \(D=(-2,4,0,4)\), \(\bar{D}=1.5\), and B has lower average loss.)
- For ten paired cases, the differences are \(1,1,0,-1,2,0,1,-2,1,0\). Calculate the mean difference and interpret its direction assuming lower loss is better. (Answer: \(\bar{D}=3/10=0.3\), favoring B on average.)
- The five observed paired differences are \(3,-1,2,0,-4\). A bootstrap replicate selects indices \(2,2,4,5\), so the replicate contains four draws. Calculate its mean difference. (Answer: \((-1-1+0-4)/4=-1.5\).)
- Four paired differences are \(2,-2,2,-2\). Calculate their sample standard deviation and the standard error of their mean using \(s_D/\sqrt{n}\). (Answer: \(\bar{D}=0\), \(s_D=\sqrt{16/3}\approx2.309\), and \(s_D/\sqrt{4}\approx1.155\).)
- Two methods have the same overall number of classification errors, but some errors occur on different cases. Which resampling unit should be used to compare their error rates, and why? (Answer: Resample held-out case indices and retain both methods’ losses for each selected case, because this preserves which methods agree or disagree on each case.)