Class Imbalance And OOB Forest Diagnostics

Why Overall OOB Error Can Be Misleading

A classification forest can make few errors overall and still fail to identify a large share of the cases in a less common class. This matters whenever the two types of cases have different practical consequences. For example, in a hypothetical model that flags insurance claims for additional review, the missed claims may matter even if they make up only a small part of all claims. A single overall error rate can conceal that weakness.

For a random forest, an observation’s out-of-bag (OOB) prediction is formed using trees whose bootstrap samples did not include that observation. The OOB classification error is the fraction of eligible observations whose OOB predicted class differs from their known class. It is a useful error estimate, but its overall value is prevalence-weighted: the more common class contributes more observations to the total.

Definition: For a binary classification problem, call one class positive and the other negative. Among the observations with OOB predictions, let \(TP\) be the number of positives predicted positive, \(FN\) the number of positives predicted negative, \(TN\) the number of negatives predicted negative, and \(FP\) the number of negatives predicted positive. The positive-class error is \(FN/(TP+FN)\); the negative-class error is \(FP/(TN+FP)\). The overall OOB error is \((FN+FP)/(TP+FN+TN+FP)\).

The positive-class error is also called the false negative rate. Its complement, \(TP/(TP+FN)\), is sensitivity or recall. The negative-class error is the false positive rate; its complement, \(TN/(TN+FP)\), is specificity. These measures answer different questions, so an OOB diagnostic should state which is being reported.

Prevalence Determines the Overall Rate

Let \(n_1=TP+FN\) be the number of positive observations and \(n_0=TN+FP\) the number of negative observations. Write \(e_1=FN/n_1\) for positive-class error, \(e_0=FP/n_0\) for negative-class error, and \(\pi=n_1/(n_1+n_0)\) for the positive-class prevalence. Then overall error is a weighted average of the two class-specific errors:

\text{Overall error} =\frac{FN+FP}{n_1+n_0} =\frac{n_1e_1+n_0e_0}{n_1+n_0} =\pi e_1+(1-\pi)e_0.

This identity follows directly from the definitions. Since \(FN=n_1e_1\) and \(FP=n_0e_0\), substitute these equalities into \((FN+FP)/(n_1+n_0)\), then divide each term in the numerator by the common denominator. The weights are the observed class proportions, not one-half each. If positives are rare, even a large positive-class error may have little effect on the overall rate.

Definition: The balanced error rate is the unweighted mean of the two class-specific error rates, \((e_1+e_0)/2\). It gives each class equal weight regardless of prevalence. It is not the ordinary overall OOB error, and it answers a different question.

A useful reference is the majority-class-only classifier, which always predicts the more common class. If the positive class is the minority, this classifier has overall error \(\pi\): it misses every positive and correctly labels every negative. It can therefore have an impressive-looking overall accuracy of \(1-\pi\) while having zero positive-class sensitivity. Compare a forest with this baseline as well as with class-specific rates; beating the baseline overall does not by itself show that minority-class performance is adequate.

Worked Examples

Worked Example: A Low Error Rate With Many Missed Positives

A hypothetical claim-review forest has OOB predictions for 1,000 claims. Of these, 40 are actually flagged claims and 960 are not. The OOB predictions produce \(TP=22\), \(FN=18\), \(TN=912\), and \(FP=48\). Calculate overall error, both class-specific errors, sensitivity, specificity, balanced error, and the majority-class-only baseline. State what the overall rate conceals.

Method. First check the row totals: \(TP+FN=22+18=40\) positives and \(TN+FP=912+48=960\) negatives. The grand total is \(40+960=1{,}000\), consistent with the data.

There are \(FN+FP=18+48=66\) OOB misclassifications, so overall error is \(66/1{,}000=0.066\), or 6.6%. Positive-class error is \(18/40=0.45\), or 45%, and sensitivity is \(22/40=0.55\), or 55%.

Negative-class error is \(48/960=0.05\), or 5%, and specificity is \(912/960=0.95\), or 95%. Balanced error is \((0.45+0.05)/2=0.25\), or 25%.

The positive prevalence is \(40/1{,}000=0.04\). A classifier that always predicts the majority negative class would make 40 errors, giving an overall error of \(40/1{,}000=0.04\), or 4%, and sensitivity of zero. The forest’s overall error of 6.6% is higher than that baseline, while its sensitivity of 55% shows that it does identify some positives. The key finding is that it misses \(18\) of the \(40\) flagged claims.

Check and trap. The errors sum to \(18+48=66\), agreeing with the overall calculation. A common mistake is to report \(1-0.066=93.4\%\) accuracy as evidence of strong performance on flagged claims. That is incorrect: accuracy combines both classes, while the observed sensitivity is only 55%.

Worked Example: Comparing Error With the Majority Baseline

A hypothetical forest is evaluated on 200 OOB-predicted records: 20 belong to a rare event class and 180 to the usual class. Its OOB confusion counts are \(TP=12\), \(FN=8\), \(TN=171\), and \(FP=9\). Calculate the overall and class-specific errors, then compare the overall result with an always-usual-class prediction.

Method. The class totals check: \(12+8=20\) rare-event records and \(171+9=180\) usual records. The forest makes \(8+9=17\) errors out of 200, so overall error is \(17/200=0.085\), or 8.5%.

Rare-event error is \(8/20=0.40\), or 40%; sensitivity is \(12/20=0.60\), or 60%. Usual-class error is \(9/180=0.05\), or 5%; specificity is \(171/180=0.95\), or 95%. Balanced error is \((0.40+0.05)/2=0.225\), or 22.5%.

The prevalence of the rare class is \(20/200=0.10\). Always predicting the usual class makes all 20 rare-event errors and no usual-class errors, for an overall error of \(20/200=0.10\), or 10%. Thus the forest improves overall error from 10% to 8.5%, but it still misses 40% of the rare events.

Check and trap. The prevalence-weighted calculation gives \(0.10(0.40)+0.90(0.05)=0.04+0.045=0.085\), matching the direct count. Do not call 22.5% the forest’s OOB error: it is the balanced error rate. The ordinary overall OOB error is 8.5%.

Worked Example: Choosing a Diagnostic for a Minority-Class Objective

Two hypothetical forests are evaluated on the same 500 OOB-predicted applications, of which 25 are in a minority class and 475 are in the majority class. Forest A has \(TP=10\), \(FN=15\), \(TN=465\), and \(FP=10\). Forest B has \(TP=17\), \(FN=8\), \(TN=455\), and \(FP=20\). Compare the forests using overall error, minority-class error, majority-class error, and balanced error. Which has better minority-class error?

Method. For Forest A, the class totals are \(10+15=25\) and \(465+10=475\). Its total errors are \(15+10=25\), so overall error is \(25/500=0.05\), or 5%. Its minority-class error is \(15/25=0.60\), and its majority-class error is \(10/475\approx0.0211\), or 2.11%. Its balanced error is \((0.60+0.0211)/2\approx0.3105\), or 31.05%.

For Forest B, the class totals are \(17+8=25\) and \(455+20=475\). Its total errors are \(8+20=28\), so overall error is \(28/500=0.056\), or 5.6%. Its minority-class error is \(8/25=0.32\), and its majority-class error is \(20/475\approx0.0421\), or 4.21%. Its balanced error is \((0.32+0.0421)/2\approx0.1811\), or 18.11%.

Forest A has the lower overall error, 5% versus 5.6%, but Forest B has the lower minority-class error, 32% versus 60%, and the lower balanced error. For an objective that emphasizes finding minority-class cases, Forest B has the better minority-class diagnostic in this comparison. The numbers alone do not determine which forest is preferable for every application; the costs of the two kinds of mistake matter.

Check and trap. The minority errors use denominator 25 in both forests; majority errors use denominator 475. Reversing those denominators would produce invalid class-specific rates. Also, the lower overall error for Forest A does not mean it is better at identifying minority cases.

Reading an OOB Confusion Table Carefully

An OOB confusion table is only meaningful when its denominators and prediction coverage are clear. For each observation, the OOB prediction is based on the trees for which that observation was left out of the bootstrap sample. In classification, those trees’ votes are aggregated to assign a predicted class; the confusion counts compare these OOB predicted classes with the known labels. The exact tie-handling rule should be specified if ties can occur.

When some observations have no OOB votes, they cannot receive an OOB prediction. Calculate error rates using the observations that do have predictions, and report the number covered and the class totals among them. Do not silently treat observations without OOB predictions as correctly classified or include them in a denominator they do not contribute to. A low-coverage class-specific rate can also be unstable, especially when the minority class contains few cases.

For more than two classes, report a confusion table and class-specific errors for each class. One common extension of balanced error is the mean of the class-specific error rates, giving each class equal weight. The ordinary overall error still counts all misclassified observations over all covered observations, so it remains influenced by class frequencies.

Key takeaway: Overall OOB error summarizes the fraction of all covered observations misclassified, weighted by class prevalence. To assess a minority class, also inspect its error rate or sensitivity, the other class’s error rate, and—when equal class weighting is useful—the balanced error rate. Compare overall error with a majority-class baseline, and always state the relevant denominators.

Common Mistakes

Equating high accuracy with good minority-class performance. Accuracy is one minus overall error; it does not reveal how errors are divided between classes.
Using the wrong denominator. The positive-class error divides false negatives by all actual positives, not by all observations. The negative-class error divides false positives by all actual negatives.
Confusing balanced error with OOB error. Balanced error weights the class-specific errors equally; ordinary OOB error weights them according to the number of observations in each class.
Ignoring the simple baseline. When one class dominates, an always-majority prediction may have low overall error. It is a reference point, not evidence that the model is useful for the minority class.
Leaving OOB coverage unstated. Error rates should be tied to the observations that actually received OOB predictions.

Check Your Understanding

For each problem, use OOB-predicted observations only. State which denominator applies before calculating a class-specific rate.

A hypothetical detector has 300 OOB-predicted cases: 30 positive and 270 negative. Its counts are \(TP=18\), \(FN=12\), \(TN=243\), and \(FP=27\). Find overall error, positive-class error, negative-class error, and balanced error. (Answer: overall error \(=39/300=13\%\); positive-class error \(=12/30=40\%\); negative-class error \(=27/270=10\%\); balanced error \(=25\%\).)
A forest has 500 OOB predictions, including 20 minority-class cases. It correctly predicts 11 minority cases and makes 24 errors among 480 majority cases. Find the minority error, majority error, and overall error. (Answer: minority error \(=9/20=45\%\); majority error \(=24/480=5\%\); overall error \(=(9+24)/500=6.6\%\).)
In a data set of 400 covered observations, 40 are positive. A forest has positive-class error \(0.30\) and negative-class error \(0.08\). Use the prevalence-weighted identity to find overall error and balanced error. (Answer: overall error \(=0.10(0.30)+0.90(0.08)=10.2\%\); balanced error \(=(0.30+0.08)/2=19\%\).)
A hypothetical forest is evaluated on 250 covered records: 25 minority and 225 majority. It has \(FN=10\) and \(FP=9\). Compare its overall error with an always-majority classifier, and find its minority-class error. (Answer: forest overall error \(=(10+9)/250=7.6\%\), versus baseline error \(=25/250=10\%\); minority-class error \(=10/25=40\%\).)
Two classes have 15 and 285 covered cases. A classifier has class-specific errors of \(0.40\) and \(0.04\), respectively. Calculate the overall error and explain why it differs from balanced error. (Answer: overall error \(=(15/300)(0.40)+(285/300)(0.04)=5.8\%\); balanced error \(=(0.40+0.04)/2=22\%\), because it gives the two classes equal rather than prevalence-based weight.)

What You'll Learn