Why OOB Coverage Needs Its Own Diagnostic
An out-of-bag (OOB) prediction for a training observation uses only trees whose bootstrap samples did not include that observation. The OOB error then summarizes how well those predictions perform. But a single OOB error value can hide an important difference: some observations may be predicted by many trees, while others may be predicted by only one or two. An estimate based on sparse coverage can be sensitive to the particular trees that happen to be available.
This tutorial focuses on diagnosing that sensitivity. It does not treat the number of OOB predictions as a formal confidence interval, nor does a large count by itself establish that a model generalizes well. The count is a first check on how much ensemble support each observation has. This diagnostic is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Expected OOB Coverage Is Not Guaranteed Coverage
For a particular observation, a single bootstrap draw has probability \(1/n\) of selecting it and probability \(1-1/n\) of not selecting it. The \(n\) draws in a bootstrap sample are independent. Therefore, the probability that the observation is absent from the whole sample is \(q=(1-1/n)^n\). If the \(B\) bootstrap samples are independently generated, the count \(K_i\) has a binomial distribution.
The expected count describes an average across repeated ensembles; it does not promise that every observation receives that many OOB predictions. The chance of no OOB prediction for a particular observation is \((1-q)^B\). More generally, the chance of \(k\) OOB trees is \(\binom{B}{k}q^k(1-q)^{B-k}\). These probabilities help identify whether a small count is surprising or an expected consequence of using a limited number of trees.
The expected number of observations with no OOB prediction is \(n(1-q)^B\), by adding the zero-coverage probabilities across the \(n\) observations. This expectation does not require the zero-coverage events for different observations to be independent. In fact, their counts are related because observations share bootstrap samples.
Worked Example: Quantifying Sparse Coverage
Worked Example: Quantifying Sparse Coverage
A hypothetical training set contains \(n=4\) observations. An ensemble uses \(B=5\) independently drawn bootstrap samples, each of size four. Calculate the expected OOB tree count for one observation and the probability that it receives no OOB prediction. Also assess the probability that it receives at most one OOB prediction.
Method. Use the binomial coverage model. The key assumption is that each tree uses four independent draws with replacement and that the five bootstrap samples are independently generated.
Step 1: Find the OOB probability for one tree. For any one observation, \(q=(1-1/4)^4=(3/4)^4=81/256=0.31640625\).
Step 2: Calculate the expected count. \(E[K_i]=Bq=5(81/256)=405/256=1.58203125\). Thus, this observation receives about 1.58 OOB trees on average across repeated ensembles, not a guaranteed count of one or two.
Step 3: Calculate the chance of no coverage. \(P(K_i=0)=(1-q)^5=(175/256)^5\approx 0.1493\). There is about a 14.93% chance that this particular observation has no OOB prediction.
Step 4: Calculate the chance of at most one OOB tree. \(P(K_i\leq 1)=P(K_i=0)+P(K_i=1)\), where \(P(K_i=1)=\binom{5}{1}(81/256)(175/256)^4\approx 0.3455\). Therefore, \(P(K_i\leq 1)\approx 0.1493+0.3455=0.4948\).
Check and mistake avoided. The probability of at most one tree includes both zero and one; reporting only the zero-coverage probability would miss almost all of the sparse-coverage risk. With these inputs, nearly half of repeated ensembles give this observation at most one OOB tree. The expected count alone would not reveal that.
Read Counts Alongside Regression Predictions
For regression, the OOB prediction for observation \(i\) is the average of the predictions from its \(K_i\) OOB trees, provided \(K_i>0\). A small \(K_i\) makes this average especially sensitive to which trees are included. One useful diagnostic is to report both \(K_i\) and the spread of the available tree predictions for selected cases, particularly cases with large OOB residuals or losses.
A sample variance can describe the spread of a case's available tree predictions when there are at least two of them. Dividing that variance by the count and taking a square root gives a familiar scale for variation in an average under an independence model. However, predictions from trees in an ensemble are not generally independent: their bootstrap samples overlap and their fitted trees use the same training set. Treat this calculation as a descriptive diagnostic, not as a justified standard error or a formal interval for the prediction.
Worked Example: Diagnosing Regression OOB Support
A hypothetical regression ensemble has OOB predictions for four observations. Their observed responses and individual tree predictions are shown below. Calculate each OOB prediction, its squared error, and the overall OOB mean squared error. Then identify which cases have especially thin support.
| Observation | Observed response | OOB tree predictions |
|---|---|---|
| A | 10 | 8, 12 |
| B | 20 | 18 |
| C | 30 | 26, 30, 34, 30 |
| D | 40 | 35, 45 |
Method. Average the available OOB predictions separately for each observation. Then square the difference between that average and the observed response. The overall OOB mean squared error is the mean of the four case-level squared errors.
Observation A. The OOB count is \(K_A=2\), and the prediction is \((8+12)/2=10\). Its error is \(10-10=0\), so its squared error is \(0^2=0\). The two predictions have sample variance \(((8-10)^2+(12-10)^2)/(2-1)=8\).
Observation B. The OOB count is \(K_B=1\), and the prediction is 18. Its error is \(18-20=-2\), so its squared error is \((-2)^2=4\). A sample variance cannot be calculated from one prediction.
Observation C. The OOB count is \(K_C=4\), and the prediction is \((26+30+34+30)/4=120/4=30\). Its squared error is \((30-30)^2=0\). The sample variance of the tree predictions is \((16+0+16+0)/(4-1)=32/3\approx 10.6667\).
Observation D. The OOB count is \(K_D=2\), and the prediction is \((35+45)/2=40\). Its squared error is \((40-40)^2=0\). The sample variance is \((25+25)/(2-1)=50\).
Overall OOB mean squared error. The four squared errors sum to \(0+4+0+0=4\), so the OOB mean squared error is \(4/4=1\).
Check and mistake avoided. Case B has only one OOB prediction, so its apparently modest squared error does not show that its OOB prediction is stable. Cases A and D each have two predictions, but their prediction spreads differ: the sample variances are 8 and 50. The aggregate error of 1 is correct for these four cases, but it conceals the variation in case-level support. Do not infer that every case is equally well assessed from the aggregate alone.
Classification Votes and Coverage
For classification, an observation's OOB prediction is commonly the class receiving the most votes from its OOB trees. The count \(K_i\) tells how many votes are available. It is useful to record the vote counts or class proportions as well: a winning class supported by three votes out of four is less decisive than one supported by a much larger set of votes with a similar proportion. If the largest vote counts are tied, the classification rule must specify how the tie is handled.
A low count can make a case's predicted class change when a small number of additional OOB trees become available. Even a large count is not a guarantee of correctness: it describes the number of votes, not whether the model has learned a suitable classification rule. OOB error also needs a clear denominator. If a case has \(K_i=0\), it has no OOB prediction; it cannot be counted as either a correct or an incorrect OOB classification. Report how many cases were covered, and do not silently treat uncovered cases as correct or remove them without stating the denominator.
Worked Example: Interpreting Sparse OOB Votes
A hypothetical classifier has four training cases, each with a nonzero OOB count. The positive class is denoted \(+\), and ties are assigned to the negative class. The true labels and OOB votes are as follows. Calculate the OOB class for each case and the OOB misclassification rate.
| Case | True class | OOB votes |
|---|---|---|
| A | + | +, −, + |
| B | − | + |
| C | + | +, − |
| D | − | −, −, −, − |
Method. For each case, count the votes for each class, apply the stated tie rule, and compare the resulting OOB class with the true class.
Case A. There are three OOB trees: two vote \(+\) and one votes \(−\). The OOB class is \(+\), which is correct.
Case B. There is one OOB tree and it votes \(+\). The OOB class is \(+\), which is incorrect because the true class is \(−\).
Case C. There are two OOB trees, one for each class. This is a tie, so the stated rule assigns \(−\). That prediction is incorrect because the true class is \(+\).
Case D. All four OOB trees vote \(−\). The OOB class is \(−\), which is correct.
OOB misclassification rate. Two of four cases are misclassified, so the rate is \(2/4=0.5\).
Check and mistake avoided. Case C is counted as an error because the tie rule assigns it to the wrong class; it is not automatically excluded. Case B's single vote also counts as a classification, but its support is particularly sparse. The error rate is \(0.5\) on the four covered cases, and the counts show why that one number is not a complete account of support.
Common Traps When Reporting Sparse-Coverage Diagnostics
- Confusing expected coverage with actual coverage. \(Bq\) is an expected count. Inspect the actual \(K_i\) values; do not assume each observation receives the expected number of OOB trees.
- Ignoring observations with zero OOB trees. An OOB prediction does not exist when \(K_i=0\). State how many cases are covered and which denominator is used for the OOB error.
- Reading an aggregate error as equally supported case by case. The same overall error can result from many well-covered cases or from a mixture that includes thinly covered cases. Check per-case counts and losses.
- Treating tree-to-tree spread as a formal confidence interval. Tree predictions share training data and are not generally independent. A spread summary is informative, but does not by itself justify a standard error or interval.
- Confusing signed deviations with absolute deviations. Suppose two loss summaries are 80 and 100, with center \((80+100)/2=90\). Their signed deviations are \(80-90=-10\) and \(100-90=10\), whose average is \((-10+10)/2=0\). Their mean absolute deviation is \((|-10|+|10|)/2=10\). The average signed deviation is 0, not 10; 10 is the mean absolute deviation.
Check Your Understanding
For each problem, calculate the requested quantity and note what the coverage does or does not establish.
- A data set has \(n=8\) cases and uses \(B=12\) independent bootstrap samples of size eight. Find the expected OOB count for one case and its probability of zero OOB trees. (Answer: \(q=(7/8)^8\approx0.3436\); \(E[K_i]\approx4.1233\); \(P(K_i=0)=(1-q)^{12}\approx0.0064\).)
- A data set has \(n=5\) cases and uses \(B=4\) bootstrap samples of size five. Find the expected OOB count and the probability of no OOB tree for a specified case. (Answer: \(q=(4/5)^5=0.32768\); \(E[K_i]=1.31072\); \(P(K_i=0)=(0.67232)^4\approx0.2043\).)
- Three regression cases have OOB residuals \(2,-1,3\), where residual means OOB prediction minus observed response. Find the OOB mean squared error. (Answer: \((2^2+(-1)^2+3^2)/3=14/3\approx4.6667\).)
- Of seven classification cases, six have OOB predictions and two of those six are misclassified. What is the covered-case OOB misclassification rate, and what coverage detail should accompany it? (Answer: \(2/6=0.3333\); report that six of seven cases were covered and one had no OOB prediction.)
- Two cases have OOB counts 1 and 9. Both have the same OOB squared error. Does that establish equally stable OOB predictions? (Answer: No. The counts show very different support; the first case's prediction is based on one tree, and its tree-to-tree variation cannot be assessed from that single prediction.)