What an OOB Error Calculation Measures
In bootstrap aggregation, each tree is fitted using a bootstrap sample. A training observation is out of bag for a particular tree when that observation was not included in the tree’s bootstrap sample. The observation’s out-of-bag, or OOB, prediction uses only trees for which it was out of bag.
The calculation has two stages. First, combine the eligible trees’ predictions separately for each observation. Then compare each resulting case-level prediction with that observation’s known response or class. The order matters: OOB mean squared error is not the average squared error of every eligible tree prediction.
The number \(m_i\) can differ across observations. Each observation nevertheless contributes one case-level error to the overall metric. For the formulas and examples below, assume every observation being evaluated has at least one eligible tree. The OOB score is calculated on those observations using their OOB predictions, not predictions from trees that included them in their bootstrap samples.
Regression: Average First, Then Square the Error
Let \(y_i\) be the observed response for case \(i\), and let \(\hat{y}_{ib}\) be the prediction for that case from eligible tree \(b\). Its regression OOB prediction and squared error are
If there are \(n\) evaluated cases, the OOB mean squared error is
The denominator is the number of evaluated cases, not the total number of eligible tree predictions. An observation with many eligible trees and an observation with few eligible trees each contribute one squared error to this average.
Worked Example: Regression OOB MSE With Unequal Tree Counts
A hypothetical severity model has three training cases. The table gives each observed response and the predictions from that case’s eligible trees. Calculate the OOB MSE. Then identify the error in pooling individual tree residuals.
| Case | Observed \(y_i\) | Eligible-tree predictions |
|---|---|---|
| 1 | 10 | 8, 12 |
| 2 | 14 | 12, 15, 18 |
| 3 | 20 | 17 |
Method. Average the eligible predictions for each case, calculate that case’s squared error, and average the three squared errors.
Case 1. There are \(m_1=2\) eligible trees:
Case 2. There are \(m_2=3\) eligible trees:
Case 3. There is \(m_3=1\) eligible tree:
OOB MSE. There are three cases, so
Rounded to four decimal places, the OOB MSE is 3.3333.
Common mistake and correction. Pooling all six tree-level squared residuals gives \((10-8)^2+(10-12)^2+(14-12)^2+(14-15)^2+(14-18)^2+(20-17)^2=38\), and \(38/6=6.3333\). That is not the OOB MSE: it scores each tree prediction separately and gives more influence to cases with more eligible trees. The required calculation first averages predictions within each case, then averages the three case-level squared errors. The result, \(10/3\), is checked by the case errors \(0,1,9\), which sum to 10.
Classification: Predict a Class, Then Count Errors
For classification, each case receives one OOB class prediction based only on its eligible trees. Under a majority-vote rule, the predicted class is the class with the most votes among those trees. If class probabilities are supplied and the prediction rule is to average probabilities, average each class’s probability across the eligible trees, then choose the class with the largest average.
Once each case has one predicted class, its error indicator is 1 if the predicted class differs from the true class and 0 otherwise. The OOB misclassification rate is the fraction of cases incorrectly classified:
where \(\mathbf{1}(\cdot)\) equals 1 when the statement in parentheses is true and 0 otherwise. A tie in votes or averaged probabilities needs a specified tie-breaking rule. State and apply that rule consistently; a tie does not create a fraction of an error.
Worked Example: Classification Votes With a Tie Rule
A hypothetical classifier predicts one of three classes, A, B, and C. Its rule is majority vote among eligible trees; when votes tie, choose the first tied class in the fixed order A, then B, then C. Calculate the OOB misclassification rate from the following case-level votes.
| Case | True class | Eligible-tree votes | OOB predicted class | Error? |
|---|---|---|---|---|
| 1 | A | A, A, A, B | — | — |
| 2 | B | A, A, B | — | — |
| 3 | C | C, A | — | — |
| 4 | A | B, B, A | — | — |
Case 1. A receives three votes and B receives one. The prediction is A, which matches the true class, so the error indicator is 0.
Case 2. A receives two votes and B receives one. The prediction is A, which differs from the true class B, so the error indicator is 1.
Case 3. A and C each receive one vote. The tie rule selects A. The true class is C, so the error indicator is 1.
Case 4. B receives two votes and A receives one. The prediction is B, which differs from the true class A, so the error indicator is 1.
There are three errors among four cases:
The OOB misclassification rate is 0.7500, or 75%. The eligible-tree counts differ, but each case supplies exactly one final class prediction and one error indicator. The tally of errors, 3, confirms the numerator.
Using Averaged Class Probabilities
Some classification procedures specify a prediction based on averaged class probabilities rather than a direct count of class labels. In that case, average probabilities within each case before selecting its predicted class. The final misclassification calculation is still based on whether that one predicted class matches the true class.
Worked Example: Classifying From OOB Probability Averages
A hypothetical three-class classifier uses the class with the largest average OOB probability as its prediction. The probability vectors below are ordered as (A, B, C). Calculate the OOB prediction for each case and the misclassification rate.
| Case | True class | Eligible-tree probability vectors |
|---|---|---|
| 1 | A | (0.7, 0.2, 0.1), (0.5, 0.3, 0.2) |
| 2 | B | (0.6, 0.3, 0.1), (0.4, 0.4, 0.2), (0.5, 0.3, 0.2) |
| 3 | C | (0.1, 0.2, 0.7) |
| 4 | A | (0.2, 0.6, 0.2), (0.3, 0.5, 0.2) |
Case 1. Average the two probabilities separately for each class:
The largest average is 0.60 for A. The prediction is A, so this case is correct.
Case 2. Average the three eligible trees’ probabilities:
The largest average is 0.50 for A. The prediction is A, not the true class B, so this case is incorrect.
Case 3. Its one eligible tree gives probabilities (0.1, 0.2, 0.7). The prediction is C, which is correct.
Case 4. The class-probability averages are \[ \left( \frac{0.2+0.3}{2}, \frac{0.6+0.5}{2}, \frac{0.2+0.2}{2} \right) = (0.25,0.55,0.20). \] The prediction is B, not the true class A, so this case is incorrect.
There are two incorrect predictions among four cases:
The misclassification rate is 0.5000, or 50%. The result is checked by counting incorrect case-level predictions: cases 2 and 4.
A Reliable Calculation Sequence
A short, consistent sequence helps prevent errors when eligible-tree counts vary or a problem provides both tree outputs and case-level information.
Use only trees for which that particular observation was out of bag. Do not assume every case has the same number of eligible trees.
For regression, average the eligible numerical predictions. For classification, apply the stated vote or averaged-probability rule, including its tie rule if needed.
For regression, square the difference between observed and OOB-predicted response. For classification, record 1 for a wrong class and 0 for a correct class.
Divide the sum of case-level scores by the number of evaluated cases. The denominator is not the total number of eligible trees.
Common Calculation Traps
- Squaring before averaging tree predictions. For regression, first calculate the case’s average OOB prediction. Squaring every tree’s separate residual produces a different quantity.
- Using a pooled tree-level denominator. A case with more eligible trees does not count as more cases. The OOB MSE and misclassification rate average case-level scores.
- Ignoring the classification rule. Use majority vote or averaged probabilities as specified. They need not produce the same class prediction.
- Leaving a tie unresolved. Apply the stated tie-breaking rule. If a problem does not supply one, state the convention being used rather than silently choosing a class.
- Including in-bag predictions. A tree contributes to case \(i\)’s OOB prediction only if case \(i\) was not in that tree’s bootstrap sample.
These calculations provide a score for the specified case-level OOB predictions. The key is to respect the two levels of the computation: aggregate eligible trees within each case, then average the resulting case-level errors.
Check Your Understanding
For each problem, calculate the requested score using the stated case-level rule. Unless noted otherwise, round rates to four decimal places.
- A regression model has two cases. For case 1, the observed response is 6 and eligible-tree predictions are 5 and 7. For case 2, the observed response is 10 and eligible-tree predictions are 8, 11, and 12. Find the OOB MSE. (Answer: \(1/18=0.0556\))
- A classifier uses majority vote among eligible trees and breaks ties in the order A, then B, then C. The true classes and votes are: case 1, A with votes A and B; case 2, B with votes A, A, and B; case 3, C with votes C, C, A, and B. Find the OOB misclassification rate. (Answer: \(1/3=0.3333\))
- A regression model has three cases: observed responses 30, 40, and 50. Their eligible-tree predictions are, respectively, (31, 29), (38, 43), and (44, 50, 56). Calculate the OOB MSE. (Answer: \(1/12=0.0833\))
- A classifier averages eligible-tree probabilities and predicts the class with the largest average. The class order is A, B, C. Case 1 has true class A and average probabilities (0.55, 0.30, 0.15); case 2 has true class C and average probabilities (0.20, 0.50, 0.30); case 3 has true class B and average probabilities (0.20, 0.40, 0.40). Ties are resolved in the order A, then B, then C. Find the misclassification rate. (Answer: \(1/3=0.3333\))
- A regression model has two cases. The first has observed response 5 and eligible predictions 4 and 6; the second has observed response 9 and eligible predictions 8, 10, and 10. Calculate the OOB MSE, then compare it with the incorrect pooled average of individual tree squared residuals. (Answer: OOB MSE \(=1/18=0.0556\); pooled tree-level average \(=1.0000\), which is not the OOB MSE.)