Class Imbalance And Classification-Tree Splits

Why Class Imbalance Matters in a Tree

A classification tree can achieve high overall accuracy while rarely recognizing the class that matters most. This can happen when one class dominates the training data: predicting the majority class for every observation may already produce a high accuracy, and a split that improves that accuracy slightly may still do little for the minority class. In actuarial applications, a hypothetical claim-screening model might have many non-fraudulent claims and few fraudulent ones. Missing a fraudulent claim and incorrectly flagging a legitimate claim may also have different consequences.

Two decisions should be kept distinct. A tree’s split rule determines which observations go to each child node. A node’s class decision assigns a predicted label to observations reaching that node. The split criterion, class prior probabilities, and misclassification costs can affect these decisions in different ways. Changing a decision threshold after fitting a tree does not, by itself, change the splits that were grown.

Definition: Class imbalance occurs when the observed or target proportions of the classes differ substantially. A class prior is the probability assigned to each class before considering a particular observation’s predictors. A misclassification cost is the penalty assigned to predicting one class when another class is true.

Impurity and the Majority-Class Effect

For a node containing observations from \(K\) classes, let \(p_k\) be the proportion in class \(k\). The Gini impurity is \(1-\sum_{k=1}^{K}p_k^2\). It is zero when the node contains only one class and larger when the class proportions are more mixed. Entropy is another commonly used impurity measure; the details of the split calculation depend on which criterion the tree-fitting method uses.

For a proposed split with left and right child nodes, the weighted child impurity is the sum of each child’s impurity multiplied by its fraction of the parent observations. The Gini improvement is the parent impurity minus that weighted child impurity. A larger positive improvement means that the split produces purer children under this criterion.

G=1-\sum_{k=1}^{K}p_k^2,\qquad \Delta G=G_{\text{parent}}- \left(\frac{n_L}{n}G_L+\frac{n_R}{n}G_R\right)

Impurity is not the same as classification accuracy. A split can improve Gini impurity even when the predicted class in a child remains the majority class. Conversely, a tree selected to maximize accuracy may overlook useful separation within the minority class. Always identify which criterion is being optimized before interpreting a split.

Worked Example: Comparing Two Splits Under Imbalance

A hypothetical training node contains 90 observations in class N and 10 in class P. Compare two candidate splits using Gini improvement. Split A creates children with counts (40 N, 0 P) and (50 N, 10 P). Split B creates children with counts (30 N, 5 P) and (60 N, 5 P). State which split has the greater Gini improvement.

Method. Calculate the parent Gini, then subtract each split’s observation-weighted child Gini. This comparison avoids judging a split only by how many observations it classifies correctly.

The parent proportions are 0.9 and 0.1, so \(G_{\text{parent}}=1-(0.9^2+0.1^2)=0.18\).

For Split A, the first child is pure, so its Gini is 0. The second child has 50 of 60 observations in N and 10 of 60 in P. Its Gini is \(1-(5/6)^2-(1/6)^2=10/36=0.2778\). The weighted child Gini is \((40/100)(0)+(60/100)(0.2778)=0.1667\), giving an improvement of \(0.18-0.1667=0.0133\).

For Split B, the first child proportions are \(6/7\) and \(1/7\), giving Gini \(1-(6/7)^2-(1/7)^2=12/49=0.2449\). The second child proportions are \(12/13\) and \(1/13\), giving Gini \(1-(12/13)^2-(1/13)^2=24/169=0.1420\). The weighted child Gini is \((35/100)(0.2449)+(65/100)(0.1420)=0.1780\), so the improvement is \(0.18-0.1780=0.0020\).

Conclusion and check. Split A has the greater improvement, approximately 0.0133 versus 0.0020. Both calculations use child sizes that add to 100, and the weighted child impurity is below the parent impurity in each case. A common mistake is to call Split B better because both children contain examples of P; the stated Gini criterion instead favors Split A.

Leaf Predictions, Priors, and Decision Thresholds

With equal costs for all errors and a rule that predicts the most frequent class, a leaf is assigned its majority class. Under that rule, a leaf containing 50 N and 10 P is labelled N, despite containing some P observations. In an imbalanced problem, this can result in weak minority-class detection. The leaf’s observed proportion of P is still useful information; the label simply compresses that information into one decision.

The observed class proportion in the training data need not be the prior appropriate for the prediction task. For example, a training sample might have been deliberately assembled with more minority-class observations than occur in the population. If class sampling is independent of predictors within each class, Bayes’ rule can adjust the estimated class probability to a different prior. This adjustment relies on that assumption: if sampling also changes the predictor distribution within a class, the simple formula is not justified.

\frac{P_{\text{target}}(P\mid x)}{P_{\text{target}}(N\mid x)} = \frac{P_{\text{sample}}(P\mid x)}{P_{\text{sample}}(N\mid x)} \cdot \frac{\pi_{\text{target}}(P)/\pi_{\text{target}}(N)} {\pi_{\text{sample}}(P)/\pi_{\text{sample}}(N)}

Here \(\pi_{\text{sample}}\) and \(\pi_{\text{target}}\) are the sample and target class priors. The equation adjusts odds, not probabilities directly. It follows from Bayes’ rule: posterior odds equal the likelihood ratio multiplied by prior odds. Under class-conditional sampling, the likelihood ratio for \(x\) is unchanged, so replacing the sample prior odds with target prior odds gives the displayed adjustment.

Worked Example: Adjusting a Leaf Probability for a New Prior

A training sample has prior probability 0.20 for class P and 0.80 for class N. In a leaf, the estimated sample probability of P is 0.25. Suppose the target population has prior probability 0.50 for P. Assume sampling depends on class but not on predictors within a class. Find the adjusted target probability of P in this leaf, then identify the majority class under equal error costs.

Method. Convert the sample posterior and both priors to odds, apply the prior-odds adjustment, then convert back to a probability. This avoids incorrectly replacing a posterior probability with the target prior.

The sample posterior odds are \(0.25/0.75=1/3\). The sample prior odds are \(0.20/0.80=1/4\), and the target prior odds are \(0.50/0.50=1\). Therefore the adjusted target odds are \((1/3)(1/(1/4))=4/3\). Odds of \(4/3\) correspond to probability \((4/3)/(1+4/3)=4/7\), or approximately 0.5714.

The adjusted probability exceeds 0.50, so equal error costs imply a P prediction. This differs from the sample probability 0.25, which would imply N under equal costs. The calculation does not establish that the target prior is actually 0.50; it uses the prior supplied in the problem.

Asymmetric costs also change the class decision, even when the estimated probability is unchanged. Let \(C_{FP}\) be the cost of predicting P when the true class is N, and \(C_{FN}\) the cost of predicting N when the true class is P. If the probability of P at a leaf is \(p\), the expected cost of predicting P is \(C_{FP}(1-p)\); the expected cost of predicting N is \(C_{FN}p\). Choose P when the first quantity is smaller.

C_{FP}(1-p)<C_{FN}p \quad\Longleftrightarrow\quad p>\frac{C_{FP}}{C_{FP}+C_{FN}}

Thus, when missing P is much more costly than falsely flagging N, the threshold for predicting P falls below 0.50. This threshold is a decision rule, not a change to the underlying probability estimate.

Worked Example: Applying Unequal Error Costs

A leaf in a hypothetical screening tree has estimated probability \(p=0.18\) of class P. A false positive costs 2 units and a false negative costs 8 units. Choose the lower-cost class prediction and calculate both expected costs.

Method. Compare expected costs for the two possible actions. This avoids using 0.50 as an automatic threshold when errors have unequal consequences.

Predicting P has expected cost \(2(1-0.18)=2(0.82)=1.64\). Predicting N has expected cost \(8(0.18)=1.44\). Since \(1.44<1.64\), predict N. Equivalently, the threshold is \(2/(2+8)=0.20\), and \(0.18<0.20\).

The two methods agree, which checks the decision. A common mistake is to predict P merely because 8 is the larger cost; the relevant comparison is expected cost, which also depends on the probability of each true class.

How Costs Can Affect Split Selection

A cost-sensitive tree can evaluate a node by the smallest expected misclassification cost available there. For a set of possible predicted actions \(a\), let \(L(a,k)\) denote the loss from predicting \(a\) when the true class is \(k\). If a node contains \(n_k\) observations of class \(k\), its minimum total empirical loss is the minimum, over actions \(a\), of the sum of \(n_kL(a,k)\). A candidate split can then be judged by the reduction in minimum loss from the parent to its two children.

R(\text{node})=\min_a\sum_{k=1}^{K}n_kL(a,k), \qquad \Delta R=R(\text{parent})-R(\text{left})-R(\text{right})

This criterion directly connects the split to the decision costs. The justification is additivity: the total cost after a split is the cost in the left child plus the cost in the right child. Subtracting that total from the parent’s minimum cost measures the reduction in empirical loss. A positive \(\Delta R\) means the split lowers that loss.

Worked Example: Evaluating a Split by Misclassification Cost

A parent node contains 90 N and 10 P observations. A false positive costs 1 unit and a false negative costs 5 units. A proposed split gives one child with 80 N and 2 P, and another with 10 N and 8 P. Use minimum empirical misclassification cost to evaluate the split, and compare it with leaving the parent unsplit.

Method. For each node, calculate the cost of predicting N and the cost of predicting P, then retain the smaller. This is a cost-sensitive split calculation; it is not the same calculation as Gini improvement.

At the parent, predicting N costs \(10(5)=50\), while predicting P costs \(90(1)=90\). The minimum parent cost is 50.

In the first child, predicting N costs \(2(5)=10\), while predicting P costs \(80(1)=80\). Its minimum is 10. In the second child, predicting N costs \(8(5)=40\), while predicting P costs \(10(1)=10\). Its minimum is 10. The children’s total cost is \(10+10=20\), so the cost reduction is \(50-20=30\).

The split reduces this empirical cost by 30 units. The check is that each child’s chosen action is the cheaper of its two action costs. The tempting but incorrect approach is to assign N to both children because N dominates the parent; the second child’s 8 P observations make predicting P cheaper under the stated costs.

A split rule such as Gini, entropy, or cost-based risk does not have identical behavior. Class priors can affect class probabilities and, in methods that use prior-weighted split scores, can affect which split is preferred. A loss matrix can affect the preferred leaf action and may also enter split evaluation in a cost-sensitive fitting procedure. In R, classification-tree implementations such as rpart provide controls for class priors, a loss matrix, and the split criterion. The precise effect depends on the implementation and settings, so inspect the fitted model and documentation rather than assuming that every option changes every stage of fitting.

Practical Checks and Exam Traps

When interpreting an imbalanced classification tree, state the class proportions, the split criterion, and the decision rule. Do not describe a tree as “good” based on accuracy alone. Consider whether the minority class is being identified and whether the relative costs of false positives and false negatives have been represented appropriately. A changed prior or cost assumption should be treated as a modeling choice with a stated rationale, not as a way to improve a score without regard to the prediction setting.

Confusing impurity with accuracy. Gini improvement measures class mixing, not the number of correct predictions under a particular threshold.
Using the majority label as the only information. A leaf’s class proportion can support a different decision if priors or costs change.
Applying a new prior without its assumption. Prior adjustment requires stable within-class predictor distributions under the sampling scheme.
Assuming a post-fit threshold changes the tree. A threshold can change predicted labels without changing the already-grown split rules.
Ignoring the loss convention. When using software, verify which matrix entries represent each true-versus-predicted error and confirm the class ordering.

Key takeaway: Class imbalance can make majority-class predictions and split choices look attractive. Separate split quality from leaf decisions, state the class priors and error costs, and calculate the criterion the tree is actually using.

Check Your Understanding

Work through each item, showing the criterion or decision rule used.

A node contains 80 N and 20 P observations. A proposed split creates two children, each containing 40 N and 10 P. What is the Gini improvement? (Answer: 0)
A leaf has 72 N and 8 P observations. Calculate its Gini impurity. (Answer: 0.1800)
A sample prior for P is 0.20, the target prior is 0.50, and the sample posterior probability of P at a leaf is 0.25. Under class-conditional sampling, what is the adjusted target probability? (Answer: \(4/7\), approximately 0.5714)
The false-positive cost is 2 and the false-negative cost is 6. At a leaf with \(P(P\mid x)=0.20\), which class minimizes expected cost, and what is the threshold for predicting P? (Answer: Predict N; the threshold is 0.25)
A parent has 45 N and 5 P. False positives cost 1 and false negatives cost 4. A split produces children (25 N, 0 P) and (20 N, 5 P). Calculate the parent minimum cost, the children’s total minimum cost, and the reduction. (Answer: Parent cost 20; children’s total cost 20; reduction 0)

What You'll Learn