Why Combine Apparent and Out-of-Bag Error?
Apparent error evaluates a classifier on the same observations used to fit it, so it can be optimistic about performance on new data. Out-of-bag error instead evaluates each bootstrap-fitted classifier on observations omitted from that bootstrap sample. It reduces this reuse of the evaluation data, but a small data set or a highly adaptable rule can still make its estimate variable or misleading. The .632 bootstrap combines the two errors, giving more weight to the out-of-bag estimate while retaining a contribution from apparent error.
The combination is a useful correction, not a theorem that guarantees an unbiased estimate of future classification risk. To understand both its construction and its limits, we first calculate how often an observation appears in an ordinary bootstrap sample.
Bootstrap Inclusion and the .632 Weight
For any one observation, a single draw misses it with probability \(1-1/n\). Independence of the \(n\) draws therefore gives the probability that it is out-of-bag. The complementary probability is its probability of appearing at least once.
As \(n\) grows, the in-bag probability approaches \(1-e^{-1}\), about \(0.632\). The out-of-bag probability approaches \(e^{-1}\), about \(0.368\). These are probabilities for a particular observation across repeated bootstrap samples; they do not mean that every bootstrap sample has exactly those proportions.
Here is the derivation for the expected fraction. Let \(I_i\) equal 1 if observation \(i\) appears at least once and 0 otherwise. The number of distinct observations in the bootstrap sample is \(\sum_{i=1}^{n} I_i\). By linearity of expectation, its expected size is the sum of the \(n\) identical inclusion probabilities. Dividing by \(n\) gives the expected fraction. Finally, the standard limit \((1-1/n)^n\to e^{-1}\) gives the stated limit.
Definition of the .632 Estimate
Let \(D_n=\{(X_i,Y_i):i=1,\ldots,n\}\) be the observed classification data, and let \(\hat g_{D_n}\) be the rule fitted to all of it. Under zero-one loss, the apparent error is the fraction of these same observations that the fitted rule misclassifies. For bootstrap replicate \(b\), fit a rule \(\hat g_{D_b}\) to bootstrap sample \(D_b\), and evaluate it only on the observations absent from that sample.
In finite samples, one may use the exact in-bag fraction \(1-(1-1/n)^n\) in place of \(0.632\), with its complement as the apparent-error weight. The conventional label and formula use the limiting rounded weights. The out-of-bag error can be computed by averaging errors over all eligible replicate-observation pairs, as defined above. A different aggregation, such as averaging replicate-specific rates equally, can give a different numerical value when the numbers of out-of-bag cases vary.
The occupancy calculation explains the characteristic weight, but does not prove that this combination estimates the target risk without bias. The .632 formula is a specified weighted correction: it pulls the typically optimistic apparent error toward an error evaluated on observations left out of a fit. The size of that correction depends on the gap between the two errors.
Worked Example: Finite-Sample Weight and Classification Error
Suppose \(n=100\), the apparent error is \(0.04\), and the out-of-bag error is \(0.20\). The exact probability that an observation is in-bag is \(1-0.99^{100}\). Since \(0.99^{100}\approx0.3660\), the exact in-bag probability is about \(0.6340\), and the out-of-bag probability is about \(0.3660\).
Using these finite-sample weights gives
Using the conventional .632 weights instead gives \(0.368(0.04)+0.632(0.20)=0.01472+0.12640=0.14112\). The small difference is due to rounding and the finite sample size. Both estimates are between the two input errors, and both are substantially closer to the out-of-bag error.
Worked Example: The Weighted Correction
A discriminant rule has apparent error \(0.08\) and out-of-bag error \(0.18\). The weighted estimate can be written as the apparent error plus a fraction of the gap:
Equivalently, \(0.14320=0.08+0.632(0.18-0.08)\). Thus the estimate corrects \(63.2\%\) of the difference between apparent and out-of-bag error. It does not take the out-of-bag error unchanged: the apparent error still receives weight \(0.368\).
What the Weighted Estimate Guarantees—and What It Does Not
To prove the result, write \(a=\hat R_{\mathrm{app}}\), \(o=\hat R_{\mathrm{OOB}}\), and \(w=0.632\). Then \(\hat R_{.632}=(1-w)a+wo\), where \(0\leq w\leq1\). If \(a\leq o\), subtracting \(a\) gives \(w(o-a)\geq0\), while subtracting \(\hat R_{.632}\) from \(o\) gives \((1-w)(o-a)\geq0\). Hence \(a\leq\hat R_{.632}\leq o\). If \(o\leq a\), the same expressions with the order reversed show \(o\leq\hat R_{.632}\leq a\). Finally, subtracting \(a\) directly gives \(\hat R_{.632}-a=w(o-a)\), as claimed. \(\square\)
This algebraic guarantee is limited: it locates the estimate between its two inputs, but says nothing about where the true future risk lies. If both inputs are optimistic, their weighted average can be optimistic too. If the out-of-bag error is highly variable, weighting it does not make that variability disappear.
Worked Example: A Memorizing Rule
Consider a binary classification problem with labels independent of the measured features and equally likely to be 0 or 1. Suppose a rule memorizes the labels of its training observations, predicting each correctly, and predicts 0 for a feature value it has never seen. On the observed data its apparent error is \(0\). On a new independent observation, the label is 1 with probability \(1/2\), so its future error is \(1/2\).
An observation left out of a bootstrap fit is unseen by this memorizing rule. Its out-of-bag prediction is 0, which is wrong with probability \(1/2\); thus the expected out-of-bag error is \(0.5\). The .632 estimate based on these values is
The estimate is below the true risk \(0.5\) by \(0.184\). This example isolates the problem: the rule has zero resubstitution error because it memorizes, and the .632 mixture retains a positive weight on that error. In an actual analysis, the numerical errors are estimated from finite data, but the same mechanism can affect a highly flexible classifier.
Limits for Discriminant Rules
The limits matter especially when a discriminant rule is unstable: small changes in the training cases can substantially change its fitted parameters, selected variables, or classifications. For example, in a high-dimensional setting with few observations, estimated covariance matrices or selected predictors can be sensitive to which cases enter a bootstrap sample. The out-of-bag predictions then reflect both the difficulty of the classification problem and the variability of the fitting procedure. A single weighted error rate does not separate these effects.
The .632 estimate can also remain too optimistic when the apparent error is extremely low because a rule nearly interpolates the training data. The memorizing-rule example demonstrates the limiting issue, not a claim that every unstable discriminant rule behaves exactly that way. Conversely, if the apparent and out-of-bag errors are close, the formula changes the apparent error only slightly; the weighting cannot repair a shared weakness in both estimates.
A further limitation is data leakage. Every step that uses outcomes or predictors to construct the classifier must be repeated inside each bootstrap fit if its performance is being assessed. If, for instance, variable selection is performed once using the full data and then held fixed in all bootstrap fits, the out-of-bag cases have already influenced the selection. They are no longer genuinely held out from the full analysis, and the resulting out-of-bag error can be optimistic.
Specify preprocessing, variable selection, parameter estimation, and any tuning that affects the final discriminant rule.
Apply that procedure using only the in-bag observations for the replicate.
Compute out-of-bag losses for observations absent from that replicate, then combine the out-of-bag error and apparent error using the stated weights.
Class balance also deserves attention. In an ordinary bootstrap, a rare class may be absent from some resamples, making a discriminant fit undefined or changing the training problem. Stratified resampling can prevent such omissions, but it changes the resampling scheme and its inclusion probabilities. The ordinary .632 weights should not be presented as an exact consequence of a differently designed bootstrap; the scheme and its rationale should be reported.
Common Mistake
Wrong reasoning: “The .632 estimate gives \(63.2\%\) weight to out-of-bag error, so it must be a nearly unbiased estimate of future risk.” The weight comes from a bootstrap inclusion probability; it is not a proof of unbiasedness. The memorizing-rule example has out-of-bag error \(0.5\), yet the weighted estimate is \(0.316\) while the future risk is \(0.5\).
Correction: Treat the .632 value as a particular compromise between apparent and out-of-bag error. Check whether the complete fitting procedure was repeated within each resample, whether the classifier is highly adaptive or unstable, and whether the two component errors are plausible measures for the intended prediction setting. Report the components as well as their weighted combination so the source of the estimate is visible.
Check Your Understanding
Use the definitions and limitations above to answer these questions.
- For a bootstrap sample of size \(n\), derive the probability that a specified observation is out-of-bag.
- A classifier has apparent error \(0.12\) and out-of-bag error \(0.22\). Calculate its conventional .632 estimate.
- Why does the fact that a .632 estimate lies between its component errors not establish that it is close to future risk?
- In the memorizing-rule example, explain why the out-of-bag error is \(0.5\) while the apparent error is zero.
- Why must variable selection that uses the outcomes be repeated within each bootstrap sample when calculating out-of-bag error?