OOB Error And Test Error

Why Three Error Measures Are Not Interchangeable

A tree ensemble can appear highly accurate on the data used to fit it and still predict new observations less accurately. Out-of-bag (OOB) error offers an internal way to assess predictive performance in bagging and random forests: each training observation is evaluated using trees whose bootstrap samples omitted that observation. OOB error is therefore different from both training error and error on an independent test set.

The key distinction is which fitted trees contribute to each prediction. A tree may be evaluated on observations used to construct it, or on observations it did not use. A separate test set goes further: its observations are held apart from the training process. These differences affect what each error measure can tell us.

Definition: In bagging, each tree is fitted to a bootstrap sample drawn with replacement from the training data. An observation is out of bag for a particular tree if it was not selected at all in that tree's bootstrap sample. Its OOB prediction combines predictions only from trees for which it was out of bag. OOB error is the average prediction loss over the training observations that have OOB predictions.

Training Error, OOB Error, and Test Error

For regression, a common loss is squared error. Let \(y_i\) be the observed response for training observation \(i\), and let \(\hat y_i\) be its prediction. Training mean squared error evaluates predictions on training observations using the fitted ensemble. Since the trees contributing to these predictions may have used those observations, training error is an apparent, in-sample measure.

For OOB evaluation, let \(S\) be the set of training observations that have at least one OOB tree and hence an OOB prediction, denoted \(\hat y_{i,\mathrm{OOB}}\). The regression OOB mean squared error is

\operatorname{MSE}_{\mathrm{OOB}} = \frac{1}{|S|} \sum_{i\in S} (y_i-\hat y_{i,\mathrm{OOB}})^2.

The denominator is \(|S|\), not automatically the full training sample size \(n\). If every training observation has an OOB prediction, then \(S\) contains all \(n\) observations and the denominator is \(n\). If some observations have no OOB trees, their OOB predictions and errors are undefined; they cannot be included in the sum.

For classification, let \(c_i\) be the observed class and \(\hat c_{i,\mathrm{OOB}}\) the OOB class prediction. With misclassification error as the loss,

\operatorname{Err}_{\mathrm{OOB}} = \frac{1}{|S|} \sum_{i\in S} I(c_i\ne \hat c_{i,\mathrm{OOB}}),

where \(I(\cdot)\) equals 1 when its statement is true and 0 otherwise. Here \(S\) consists of observations with OOB class predictions. Other losses, such as a loss based on predicted class probabilities, may be used when appropriate, but the loss definition must be stated.

An independent test error is calculated on a separate test sample not used to fit the model. It estimates predictive performance on new observations from the target population, subject to sampling variability and the assumption that the test observations represent that population. It is not itself the exact population prediction error.

Measure	Observations used for evaluation	Main interpretation
Training error	Training observations, often predicted using the full fitted ensemble	Apparent fit; often optimistic as a measure of prediction on new data
OOB error	Training observations predicted only by trees that omitted them	Internal estimate of predictive performance from bootstrap resampling
Test error	Separate observations held out from model fitting	Evaluation on a sample not used to fit the model

Why an Observation Is Out of Bag

Suppose a bootstrap sample contains \(n\) draws from \(n\) training observations, with replacement. For a particular observation, the probability of missing it on any one draw is \(1-1/n\). Because the draws are independent, the probability that it is missed on all \(n\) draws is

\left(1-\frac{1}{n}\right)^n.

This is the probability that the observation is out of bag for one tree. As \(n\) grows, this probability approaches \(e^{-1}\), about 0.368. Thus, each observation is typically available to assess a substantial fraction of the trees in a large ensemble. This probability does not guarantee that every observation receives an OOB prediction in a finite ensemble.

Interpretation: OOB error estimates prediction error for an ensemble built from bootstrap samples, using predictions made by trees that did not use the evaluated observation. Under appropriate sampling conditions, it is a practical internal estimate of performance on new data. It is not identical to the error of a single tree, training error, or independent test error.

Worked Examples

Worked Example: Comparing Regression Errors

A hypothetical insurer models the cost of a particular service using a bagged regression tree. For four training observations, the observed responses are \(10, 14, 9, 17\). The fitted ensemble's training predictions are \(10, 13, 9, 16\), and its OOB predictions are \(12, 13, 8, 20\). On five separate test observations, actual responses are \(11, 15, 7, 18, 13\), and predictions are \(10, 13, 9, 16, 15\). Calculate the training, OOB, and test MSEs. The OOB predictions are available for all four training observations.

Method. For each data set, square each actual-minus-predicted residual, add the squared residuals, and divide by the number of observations in that set.

Training error. The residuals are \(10-10=0\), \(14-13=1\), \(9-9=0\), and \(17-16=1\). The squared residuals are \(0,1,0,1\), with sum \(2\). Therefore, the training MSE is \(2/4=0.5\).

OOB error. The OOB residuals are \(10-12=-2\), \(14-13=1\), \(9-8=1\), and \(17-20=-3\). The squared residuals are \(4,1,1,9\), with sum \(15\). All four observations have OOB predictions, so \(|S|=4\), and the OOB MSE is \(15/4=3.75\).

Test error. The test residuals are \(11-10=1\), \(15-13=2\), \(7-9=-2\), \(18-16=2\), and \(13-15=-2\). Their squares are \(1,4,4,4,4\), with sum \(17\). The test MSE is \(17/5=3.4\).

The three errors are \(0.5\), \(3.75\), and \(3.4\), respectively. The smaller training error is consistent with training predictions being evaluated on observations used in fitting. The OOB and test values are close here, but that is not guaranteed. The common mistake avoided is treating training error as if it were an independent estimate of new-data performance.

Worked Example: Comparing Classification Errors

A hypothetical tree ensemble classifies whether a claim requires a specialist review. For six training claims, the actual classes are \(A,A,A,B,B,B\). The full ensemble predicts \(A,A,A,B,B,B\) on these training observations, while the OOB class predictions are \(A,B,A,A,B,B\). On a separate test set of eight claims, two predictions are incorrect. Calculate the three misclassification errors.

Method. Misclassification error is the number of incorrect class predictions divided by the number of observations evaluated.

Training error. All six full-ensemble predictions match their actual classes, so the error is \(0/6=0\).

OOB error. The second observation has actual class \(A\) but OOB prediction \(B\). The fourth has actual class \(B\) but OOB prediction \(A\). The other four OOB predictions are correct. Thus, the OOB error is \(2/6=1/3\), or approximately \(0.3333\).

Test error. Two of the eight independent test predictions are incorrect, so the test error is \(2/8=0.25\).

The zero training error does not establish perfect predictive performance: the OOB and test errors are both positive. The OOB and test values differ because they come from different evaluation observations and prediction procedures. The common mistake avoided is interpreting zero training misclassification as proof that new cases will always be classified correctly.

What OOB Error Estimates—and What It Does Not

For an observation \(i\), its OOB prediction is formed without using \(i\) to fit any of the contributing trees. This is the central reason OOB error is more informative about prediction on new observations than ordinary training error. However, the OOB prediction is based on only the subset of trees that omitted \(i\), whereas the final ensemble prediction may use all fitted trees.

There is also a training-size distinction. Each bootstrap sample contains \(n\) draws, but it typically contains fewer than \(n\) distinct observations. An OOB tree is fitted without the observation being evaluated, and its bootstrap sample includes a variable number of distinct observations from the remaining data. Consequently, OOB error is a useful estimate for the bagging procedure, not an exact measurement of the performance of a model trained on a separate sample of exactly the same size as the full training set.

OOB evaluation is most directly useful when observations are sampled independently from the same population and the bootstrap resampling scheme matches the data structure. If observations are clustered, repeated measures from the same person, or ordered in time, resampling individual rows independently can break their dependence or ignore time ordering. In such settings, the resampling and evaluation design must respect the sampling structure; an ordinary row-wise OOB calculation may not estimate the intended predictive performance.

The OOB set is also not a permanently untouched test set. If a modeller repeatedly tries different predictors, tree settings, or modeling choices and selects the version with the smallest OOB error, the OOB results have influenced model selection. The selected error can then be optimistic. A final independent test set is useful when an evaluation is needed after all model choices have been made.

Fit bootstrap trees.
Draw bootstrap samples from the training data and fit one tree to each sample.

Identify omitted observations.
For each training observation, find the trees whose bootstrap samples did not contain it.

Form OOB predictions.
Combine only those trees' predictions for each observation that has at least one such tree.

Calculate and interpret the loss.
Average the specified loss over observations with OOB predictions, then interpret it as an internal estimate rather than a guaranteed test error.

Worked Example: Handling an Observation Without an OOB Prediction

A small hypothetical data set has five training observations and uses three bootstrap trees. For each tree, a bootstrap sample consists of five draws from the five observations. The probability that a particular observation is OOB for one tree is \((4/5)^5=1024/3125=0.32768\). The probability that it is included in at least one of the three trees' bootstrap samples is not the quantity needed here; the probability it has no OOB tree is the probability it is in-bag for all three trees.

For one tree, the probability of being in-bag is \(1-0.32768=0.67232\). Assuming the three bootstrap samples are drawn independently, the probability of no OOB tree is \(0.67232^3\), approximately \(0.3039\). Thus, a finite ensemble can leave an observation without an OOB prediction.

In one realized ensemble, suppose four observations have OOB squared errors \(0.25,1,2.25,4\), and the fifth has no OOB tree. The sum of the defined squared errors is \(0.25+1+2.25+4=7.5\), and \(|S|=4\). Therefore,

\operatorname{MSE}_{\mathrm{OOB}}=\frac{7.5}{4}=1.875.

The fifth observation is not assigned an OOB squared error; it is not counted as a zero-error case. Dividing \(7.5\) by five would give \(1.5\), but that would incorrectly treat the undefined error as zero. The correct denominator is the number of observations with OOB predictions. This example is a small-ensemble illustration; a practical estimate generally benefits from enough trees to provide stable OOB predictions.

Common Mistakes and Practical Use

Using all \(n\) observations automatically in the OOB denominator. Use \(|S|\), the number with OOB predictions. The denominator is \(n\) only when \(S\) contains every training observation.
Calling OOB error training error. The responses are from the training data, but each OOB prediction uses trees that omitted that observation. That makes OOB evaluation distinct from ordinary in-sample evaluation.
Claiming OOB error must equal test error. Both can estimate predictive performance, but they use different observations and procedures and can differ due to sampling variability and training-size effects.
Tuning repeatedly against OOB error and treating the selected value as untouched. Repeated selection uses the OOB results and can make the reported minimum optimistic; use a separate test set for a final independent evaluation when needed.
Ignoring the data structure. Row-wise bootstrap evaluation is not automatically appropriate for dependent observations, such as repeated records from one individual or time-ordered data.

Key takeaway: OOB error is an internal estimate based on predictions from trees that omitted the evaluated training observations. Calculate it only over observations with OOB predictions, distinguish it from apparent training error and independent test error, and interpret it in light of the sampling design and any model selection.

Check Your Understanding

For each problem, identify the appropriate error calculation and interpret what it measures.

A regression ensemble has OOB predictions for three of four training observations. Their squared errors are \(1,4,4\); the fourth observation has no OOB tree. What is the OOB MSE? (Answer: \(9/3=3\), using the three observations with OOB predictions.)
A classifier makes 5 incorrect OOB predictions among 25 observations with OOB class predictions. What is its OOB misclassification error? (Answer: \(5/25=0.20\).)
For \(n=4\), what is the probability that a specified observation is out of bag for one bootstrap sample of size four? (Answer: \((3/4)^4=81/256\approx0.3164\).)
A regression model has training MSE \(0.8\), OOB MSE \(2.1\), and independent test MSE \(2.4\). Which value is the apparent in-sample error, and which two values assess prediction beyond ordinary training fit? (Answer: \(0.8\) is training error; \(2.1\) is OOB error and \(2.4\) is test error.)
A modeler tries many ensemble settings and reports the smallest OOB error among them as a final, unbiased performance estimate. What is the concern, and what evaluation can help address it? (Answer: Repeated selection using OOB error can make the selected value optimistic; evaluate the chosen model on a separate test set that was not used for fitting or selection.)

What You'll Learn