Out-Of-Bag Predictions For Individual Cases

Why an Out-of-Bag Prediction Is Case-Specific

A random forest is built from many trees, and each tree is fitted using a bootstrap sample: a sample of the training rows drawn with replacement. As a result, some observations appear in a tree’s bootstrap sample and some do not. For a particular observation, only the trees whose bootstrap samples omitted it can contribute to its out-of-bag (OOB) prediction.

This makes an OOB prediction case-specific. Two observations in the same training data set may have different eligible tree sets, and therefore different numbers of tree predictions to combine. The procedure is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: For observation $i$, a tree is out-of-bag for $i$ if observation $i$ does not appear in that tree’s bootstrap sample. Let $B_i$ denote the set of trees out-of-bag for $i$, and let $m_i=|B_i|$ be the number of such trees. An OOB prediction for observation $i$ combines only the predictions from trees in $B_i$.

A row is in-bag if it appears at least once in the bootstrap sample; repeated appearances still count as in-bag. The test is whether the row was included, not how many times it was drawn. For the calculations below, assume each tree uses a bootstrap sample of the same size as the training set.

Constructing the Prediction

For regression, each eligible tree produces a numerical prediction for the observation. The OOB regression prediction is the arithmetic mean of those predictions. For classification, there are two common summaries: a majority vote among the eligible trees’ predicted classes, or an average of their predicted class probabilities. These are distinct procedures and need not produce the same result.

\widehat{y}_{i,\mathrm{OOB}} = \frac{1}{m_i}\sum_{b\in B_i}\widehat{y}_{ib}, \qquad m_i>0

Here $\widehat{y}_{ib}$ is the prediction for observation $i$ from tree $b$. For classification, let $\widehat{c}_{ib}$ be tree $b$’s predicted class. The vote proportion for class $k$ is the number of eligible trees predicting $k$, divided by $m_i$. The majority-vote prediction is the class with the greatest vote proportion. If trees provide class probabilities $p_{ibk}$, the average OOB probability for class $k$ is the mean of those probabilities across $B_i$; the predicted class is the one with the largest average probability.

\widehat{P}_{i,\mathrm{OOB}}(Y=k) = \frac{1}{m_i}\sum_{b\in B_i}p_{ibk}, \qquad m_i>0

When a tree outputs probabilities, the hard class it predicts is usually the class with the largest probability for that tree. Averaging those hard class labels as though they were numerical values is not meaningful: labels have no numerical scale. State whether a classification result is based on votes or averaged probabilities, and apply that convention consistently.

Why an Observation Is Often Omitted

The chance that one training observation is omitted from a bootstrap sample can be derived directly. If there are $n$ training rows and $n$ independent draws with replacement, the probability that one draw is not the specified row is $1-1/n$. The observation is omitted only if this happens on all $n$ draws.

P(\text{omitted from one tree}) = \left(1-\frac{1}{n}\right)^n \longrightarrow e^{-1}\approx 0.368

Thus, for a large training set, a tree omits a particular row about 36.8% of the time on average. With $T$ independently generated trees, the expected number of OOB trees for that observation is $T(1-1/n)^n$. This is an expectation, not a guarantee: an individual observation can have more or fewer OOB trees, including none in a small forest.

Justification: Each of the $n$ bootstrap draws independently misses the specified observation with probability $1-1/n$. Multiplying those $n$ probabilities gives $(1-1/n)^n$. The limit as $n$ grows is $e^{-1}$, so the observation’s OOB status must be checked tree by tree rather than assumed from an average percentage.

Worked Predictions

Worked Example: OOB Prediction for a Claim Cost

A hypothetical training set has six policies. A five-tree regression forest draws six policies with replacement for each tree. For policy A, the bootstrap records show that trees 1, 3, and 5 omitted A, while trees 2 and 4 included A. The eligible trees predict claim costs of $1,280, $1,460, and $1,320, respectively. Find A’s OOB prediction.

Method. Identify $B_A=\{1,3,5\}$, then average only those trees’ predictions:

\widehat{y}_{A,\mathrm{OOB}} = \frac{1{,}280+1{,}460+1{,}320}{3} = \frac{4{,}060}{3} = 1{,}353.333\ldots

The OOB prediction is $1,353.33, rounded to the nearest cent. As a check, the numerator is the sum of three eligible predictions, and dividing by three gives a value between the smallest ($1,280) and largest ($1,460) included values.

Common mistake avoided. Averaging all five predictions would include trees 2 and 4, which used policy A in fitting. Those predictions do not belong in A’s OOB calculation.

Worked Example: OOB Classification by Votes and Probabilities

A hypothetical classification forest predicts whether a policy should receive a high-risk review. For policy B, trees 1, 3, 4, and 5 omitted B; tree 2 included it and is excluded. The eligible trees’ probabilities of “High” are 0.20, 0.65, 0.35, and 0.10. Each tree predicts the class with the larger of its “High” and “Standard” probabilities. Find the vote result and the average-probability result.

Step 1: Determine each eligible tree’s class. Tree 1 predicts Standard because 0.20 is below 0.50. Tree 3 predicts High because 0.65 exceeds 0.50. Tree 4 predicts Standard because 0.35 is below 0.50. Tree 5 predicts Standard because 0.10 is below 0.50.

Step 2: Calculate the vote proportions. There are three Standard votes and one High vote among four eligible trees. The Standard vote proportion is $3/4=0.7500$, and the High vote proportion is $1/4=0.2500$. The majority-vote class is Standard.

Step 3: Average the probabilities.

\widehat{P}_{B,\mathrm{OOB}}(Y=\text{High}) = \frac{0.20+0.65+0.35+0.10}{4} = \frac{1.30}{4} = 0.3250

The average probability of Standard is $1-0.3250=0.6750$, so the average-probability method also predicts Standard. The two methods agree here, but this is not guaranteed.

Check. The two averaged class probabilities sum to 1.0000. Tree 2 contributes to neither calculation because it included policy B. The common mistake is to average all five tree probabilities; that would use an in-bag prediction.

Worked Example: Different Cases, Different OOB Sets

A hypothetical regression forest has five trees. For two training observations, the recorded omission status and tree predictions are as follows. A dash means that the tree included the observation, so its prediction is not used for that observation’s OOB result.

Tree	Observation C: omitted?	Prediction for C	Observation D: omitted?	Prediction for D
1	Yes	12	No	—
2	Yes	18	Yes	8
3	No	—	Yes	10
4	No	—	No	—
5	Yes	15	No	—

For C, $B_C=\{1,2,5\}$. Therefore,

\widehat{y}_{C,\mathrm{OOB}} = \frac{12+18+15}{3} = \frac{45}{3} = 15

For D, $B_D=\{2,3\}$. Therefore,

\widehat{y}_{D,\mathrm{OOB}} = \frac{8+10}{2} = \frac{18}{2} = 9

C and D use different tree sets and different denominators: C has three eligible predictions and D has two. Each result is checked by confirming that its sum includes only the values in rows marked “Yes.” The common mistake is to use the same tree set or denominator for every case simply because the forest is shared.

Important Boundaries and Common Traps

An OOB prediction exists only when $m_i>0$. If every tree included observation $i$, there is no eligible tree output to average or vote. Do not silently substitute the full-forest prediction: that prediction includes trees fitted using the observation. A forest with more trees will generally provide more OOB predictions, but the omission status must still be checked for the case at hand.

The OOB calculation also depends on using the tree’s prediction for the case, not the observed response. For example, when predicting a claim cost, average the eligible trees’ predicted costs; do not add the actual claim cost to the numerator. The observed response is not a tree prediction.

Wrong denominator: divide by the number of OOB trees, not the total number of trees.
Wrong inclusion rule: a case is in-bag if it was drawn at least once, even if it was drawn repeatedly.
Wrong classification calculation: specify votes or averaged probabilities; do not treat class labels as numeric quantities to average.
Wrong assumption about cases: each observation has its own OOB tree set, so counts and predictions may differ across rows.

Key takeaway: For each observation, first identify exactly which trees omitted it. Then combine only those trees’ outputs, using a mean for regression and a clearly specified vote or probability rule for classification. If the eligible set is empty, no OOB prediction is defined.

Check Your Understanding

For each problem, use only the trees that omitted the specified observation. Answers are given so you can check both the eligible set and the calculation.

A regression forest has six trees. For one case, trees 1, 4, and 5 omitted it and predicted 24, 36, and 30. Find its OOB prediction. (Answer: 30)
For a two-class outcome, trees 2, 3, and 6 omitted a case and predicted classes A, B, and B. Find the majority-vote class and its vote proportion. (Answer: Class B, with vote proportion $2/3=0.6667$)
Three eligible classification trees give probabilities of “Yes” equal to 0.15, 0.55, and 0.70. Find the average “Yes” probability and the class from averaged probabilities. Also find the majority-vote class using a 0.50 threshold for each tree. (Answer: Average “Yes” probability $=1.40/3=0.4667$, so averaged probabilities predict No; tree-level votes are No, Yes, Yes, so majority vote predicts Yes)
For case A, only trees 1 and 3 are OOB and predict 100 and 160. For case B, only trees 2, 3, and 4 are OOB and predict 80, 110, and 140. Find each case’s OOB regression prediction. (Answer: Case A $=130$; case B $=110$)
A case was included in every tree’s bootstrap sample. Can its OOB prediction be calculated from those trees? (Answer: No. Its OOB tree set is empty, so the OOB prediction is undefined.)

What You'll Learn