What an OOB Error Measures
A random forest creates each tree from a bootstrap sample of the training observations. For any one observation, the trees whose bootstrap samples omit it can provide an out-of-bag (OOB) prediction. Comparing that prediction with the observation's known outcome gives an estimate of predictive error without setting aside a separate validation sample.
The key qualification is that OOB error is an estimate, not a guarantee of future performance. Its interpretation depends on how predictions are aggregated, whether each case receives enough OOB trees, and whether decisions about the model have already been influenced by the OOB results. The earlier material on bootstrap sampling and OOB predictions provides the sampling setup; here the focus is on interpreting the resulting error carefully.
For a regression forest, the OOB prediction for case \(i\) is typically the mean of that case's predictions from its OOB trees. If \(y_i\) is the observed outcome and \(\hat{y}_{i,\mathrm{OOB}}\) is this prediction, the case-level OOB mean squared error (MSE) and root mean squared error (RMSE) are
Here \(n_{\mathrm{OOB}}\) is the number of observations with an OOB prediction. If every case has one, it equals the training size \(n\). For classification, OOB tree predictions are combined for each case, commonly by voting or by averaging class probabilities. A classification error rate is then the proportion of cases whose predicted class differs from the observed class. A probability threshold and its rule for equality must be specified when classes are assigned from probabilities.
Why OOB Coverage Varies
Suppose a bootstrap sample draws \(n\) times, with replacement, from \(n\) training observations. A particular case is omitted if every draw selects one of the other \(n-1\) cases. Its probability of being omitted from one tree is therefore
The approximation becomes more accurate as \(n\) grows. With \(B\) independently generated bootstrap samples, the number of OOB trees for a fixed case has a binomial distribution with parameters \(B\) and \(q_{\mathrm{OOB}}\). Its expected count is \(Bq_{\mathrm{OOB}}\), and the probability that the case has no OOB trees is \((1-q_{\mathrm{OOB}})^B\). Thus, each case's OOB prediction may be based on a different number of trees, especially when \(B\) is small.
This variation matters because predictions based on few trees may be noisier than predictions based on many. More trees generally reduce this source of Monte Carlo variability, but a low overall OOB error does not itself show that every case was predicted with equal precision. Always check whether the forest produced OOB predictions for all the observations included in the error calculation.
Worked Example: Regression OOB Error
A hypothetical regression forest produces the following OOB predictions for three training cases. Calculate the OOB MSE and RMSE, retaining the units of the outcome.
| Case | Observed outcome | OOB prediction | Squared residual |
|---|---|---|---|
| 1 | 12 | 10 | 4 |
| 2 | 18 | 21 | 9 |
| 3 | 20 | 19 | 1 |
Method. Compute each observed-minus-predicted residual, square it, average the three squared residuals, and take the square root for RMSE.
The residuals are \(12-10=2\), \(18-21=-3\), and \(20-19=1\). Their squares are \(2^2=4\), \((-3)^2=9\), and \(1^2=1\). Therefore
The RMSE is \(\sqrt{14/3}\approx 2.160\) outcome units, rounded to three decimal places. As a check, the RMSE is the square root of the MSE and is smaller than the largest absolute residual, which is 3. The common mistake to avoid is reporting MSE \(=3\): the three squared residuals sum to 14, not 9, so their mean is \(14/3\), not 3.
Calculate Error Across Cases, Not Trees
The OOB prediction belongs to a case: it combines the predictions from the trees that omitted that case. The usual case-level error is then calculated once for each case and averaged across cases. This is not generally the same as calculating an error for each tree on the cases omitted by that tree and averaging those tree-level errors. A tree-level average can give different weight to cases depending on how often they were omitted, and it does not use the same case-level aggregation as the forest prediction.
For classification, a probability-based OOB prediction needs a decision rule before a misclassification rate can be calculated. For example, with a two-class outcome coded 0 or 1, one may assign class 1 when the OOB probability is at least 0.50 and class 0 otherwise. The threshold is a modeling choice, not a property of OOB sampling.
Worked Example: OOB Classification Error at a Stated Threshold
A hypothetical classification forest gives the following probabilities of class 1. Use a threshold of 0.50, assigning equality to class 1, and calculate the number of errors and the OOB error rate.
| Case | Observed class | OOB probability of class 1 | Predicted class | Error? |
|---|---|---|---|---|
| 1 | 1 | 0.52 | 1 | No |
| 2 | 0 | 0.48 | 0 | No |
| 3 | 1 | 0.50 | 1 | No |
| 4 | 0 | 0.70 | 1 | Yes |
Method. Apply the stated threshold to each probability, then compare the resulting predicted class with the observed class. Cases 1 and 2 are classified correctly. Equality is assigned to class 1, so case 3 is also correct. Case 4 is assigned class 1 despite having observed class 0, so it is an error.
There is one error among four cases. The OOB error rate is \(1/4=0.2500\), or 25.00%. A tempting but incorrect calculation is to count the 0.50 probability as an error because it is “on the boundary,” then report two errors. The stated rule assigns that case to class 1, and its observed class is 1; it is correct. The threshold convention changes the classification here, so it must be applied exactly as given.
Why the OOB Construction Works—and What It Does Not Guarantee
The basic justification is that a tree used to predict case \(i\) for its OOB prediction was fit without case \(i\) in its bootstrap sample. This prevents that tree from directly fitting the response of case \(i\). The averaging step uses only such trees for that case. Consequently, under suitable conditions, OOB predictions can approximate predictions for new observations from the same population.
The bootstrap calculation explains the chance of this omission: each of the \(n\) draws has probability \(1-1/n\) of not selecting case \(i\), and the draws are independent. Multiplying these probabilities gives \((1-1/n)^n\). This exclusion is the source of the OOB assessment; it is not the same as predicting a case with a tree that included that case.
However, OOB prediction does not automatically make the entire modeling process independent of the case being assessed. For example, suppose a data-dependent preprocessing step uses all outcomes to select predictors before the bootstrap samples are drawn. A case may influence the selected predictors even when its outcome is excluded from a particular tree. The same concern applies if analysts repeatedly try model specifications, thresholds, or other choices and retain the one with the smallest OOB error. In both situations, the reported OOB result has been influenced by information from the cases it is meant to assess.
For each observation, aggregate predictions only from trees whose bootstrap samples omitted that observation.
Compute case-level regression losses or classification errors using a clearly stated threshold or decision rule.
Check whether preprocessing, feature selection, thresholds, or model selection used the same OOB results or outcomes.
Interpret the value as an estimate under the fitting and selection process used, not as a guaranteed future error rate.
Worked Example: How Adaptive Selection Can Look Optimistic
To isolate the effect of selection, suppose two candidate forest procedures have the same true error rate of 0.20. For each candidate, imagine its OOB estimate is either 0.16 or 0.24, each with probability 0.50. Assume the two estimates vary independently. An analyst chooses whichever candidate has the smaller OOB estimate. Find the expected reported error of the selected candidate.
Method. Enumerate the possible pairs of OOB estimates. The pair is (0.16, 0.16) with probability 0.25; (0.16, 0.24) with probability 0.25; (0.24, 0.16) with probability 0.25; and (0.24, 0.24) with probability 0.25. The selected minimum is 0.16 in the first three cases and 0.24 in the last case.
Each estimate separately has expected value \(0.50(0.16)+0.50(0.24)=0.20\), matching the assumed true error. After selecting the smaller estimate, the expected reported value is 0.18, below 0.20. In this simplified illustration, selection makes the winning OOB score optimistic on average. Real candidate estimates need not be independent or have these probabilities, so the amount of optimism cannot be inferred from this calculation; the mechanism is what matters.
The common mistake is to treat the selected minimum as though it had been specified before the OOB results were seen. If a modeling decision is chosen using OOB performance, an untouched test set or a suitable validation process is needed for an assessment that accounts for that selection. The OOB score can still be useful for exploration, but it is no longer a neutral final assessment of the entire selection process.
Interpreting an OOB Result in Practice
Even when the OOB procedure is otherwise appropriate, several features limit how precisely one number describes future performance. First, the OOB trees for a case are trained on bootstrap samples that omit that case. The resulting effective training data differ from the full data used to fit a final forest. This difference can make OOB performance differ from full-fit performance; its direction is not guaranteed. Second, OOB predictions for different cases share many trees and training observations, so the individual case errors should not be treated as independent measurements when making informal precision claims.
Third, a small data set may produce a noisy error estimate, especially if the outcome is rare or if cases with no OOB predictions are dropped. Fourth, a misclassification rate depends on the chosen threshold and may conceal important differences in error types. A useful report therefore names the metric, the threshold when relevant, the number of cases included, and any model choices made after reviewing OOB results.
Common Mistakes
- Confusing MSE with RMSE. MSE averages squared residuals and is in squared outcome units; RMSE is its square root and is in outcome units.
- Ignoring the classification rule. A probability equal to the threshold must be classified according to the stated equality convention.
- Averaging tree-level errors as if they were case-level OOB error. First aggregate eligible tree predictions for each case, then calculate the specified metric across cases.
- Calling a selected OOB minimum a final independent estimate. Repeatedly using the score to choose among alternatives can make the reported winner optimistic.
- Assuming every case has equal OOB support. The number of trees omitting a case varies, and may be zero when the forest has too few trees.
Check Your Understanding
For each problem, apply the stated OOB rule and check whether the result supports the conclusion being drawn.
- A regression forest has observed outcomes 8, 11, and 14, with OOB predictions 9, 9, and 16. Calculate OOB MSE and RMSE. (Answer: MSE \(=3\); RMSE \(=\sqrt{3}\approx 1.732\) outcome units.)
- Four observed classes are \(0,1,1,0\), and OOB probabilities of class 1 are \(0.49,0.50,0.31,0.51\). Use threshold 0.50, assigning equality to class 1. Find the number of errors and the OOB error rate. (Answer: 2 errors; OOB error \(=2/4=0.5000\).)
- For a large training set, use \(q_{\mathrm{OOB}}\approx0.368\). A forest has \(B=5\) independently generated trees. Approximate the expected number of OOB trees for a case and the probability it has no OOB trees. (Answer: Expected count \(=5(0.368)=1.84\); probability of none \(\approx(1-0.368)^5=0.632^5\approx0.1008\).)
- Two procedures have the same true error of 0.15. Each OOB estimate is independently 0.12 or 0.18 with equal probability, and the lower estimate is selected. Calculate the expected selected estimate. (Answer: \(0.75(0.12)+0.25(0.18)=0.135\), below the common true error of 0.15.)
- An analyst tries several preprocessing and model choices, keeps the combination with the smallest OOB error, and reports that same minimum as final performance. Is the reported minimum an independent assessment of the selection process? (Answer: No; using OOB results to select the combination can make the selected minimum optimistic, so use an untouched assessment for final evaluation.)