Why Class Imbalance Matters in a Tree
A classification tree summarizes the classes represented in each node and uses those summaries to choose splits and make predictions. When one class is rare, the overall majority can dominate both the node's predicted label and its impurity calculation. A tree can therefore have high accuracy while failing to identify many members of the less common class. This is important in hypothetical applications such as flagging unusual claim patterns or identifying a rare equipment failure: missing the rare class may matter even when most classifications are correct.
Class imbalance does not, by itself, prove that a tree is inadequate or that class weights must be used. The appropriate treatment depends on the prediction objective. First distinguish the observed class proportions from any weights applied during tree fitting. Then calculate what the split criterion and node prediction actually use.
For two classes, the Gini impurity of a node is \(G=1-p^2-(1-p)^2=2p(1-p)\). It is zero when the node contains only one class and reaches \(0.5\) when the classes are equally represented. Entropy is \(H=-p\log_2(p)-(1-p)\log_2(1-p)\), with a term defined as zero when its proportion is zero. Both measures describe class mixture, not the cost of a particular mistake.
A majority-class label and a useful model are not the same thing. If 97 of 100 observations belong to class 0, always predicting class 0 gives 97% accuracy on those observations, yet it identifies none of the three class 1 cases. Accuracy reports the fraction classified correctly; it does not show how performance is distributed across classes.
Worked Example: A Rare Class and a Split With Little Gini Gain
A hypothetical node contains 4 class 1 observations and 96 class 0 observations. A proposed split creates a left child with 3 class 1 and 17 class 0 observations, and a right child with 1 class 1 and 79 class 0 observations. Calculate the parent and child Gini impurities, the weighted split gain, and the parent majority prediction. State what accuracy alone would suggest.
Method. Use raw proportions, calculate the Gini impurity in each node, and weight each child impurity by its fraction of the 100 observations. This avoids treating two children of different sizes as equally important.
Parent. The class 1 proportion is \(4/100=0.04\), so \(G_{\text{parent}}=2(0.04)(0.96)=0.0768\). The parent predicts class 0 because \(96>4\). Its majority-only accuracy is \(96/100=0.96\), but it would miss all four class 1 cases.
Left child. Its class 1 proportion is \(3/20=0.15\), giving \(G_L=2(0.15)(0.85)=0.255\).
Right child. Its class 1 proportion is \(1/80=0.0125\), giving \(G_R=2(0.0125)(0.9875)=0.0246875\).
Weighted child impurity and gain. The child sizes are 20 and 80, so \(G_{\text{children}}=(20/100)(0.255)+(80/100)(0.0246875)=0.051+0.01975=0.07075\). Thus the Gini gain is \(0.0768-0.07075=0.00605\). The split reduces impurity, but the reduction is small.
Check. The children contain \(3+1=4\) class 1 observations and \(17+79=96\) class 0 observations, matching the parent counts. The split gain is positive, as expected from the calculated reduction in weighted impurity. A common mistake is to conclude that 96% accuracy means the rare class is being handled well; here the majority-only rule misses every class 1 observation.
Weighted Impurity and Weighted Predictions
Class weighting assigns a nonnegative weight to each observation according to its class. Let \(w_1>0\) be the class 1 weight and \(w_0>0\) the class 0 weight. In a node, the weighted class masses are \(w_1n_1\) and \(w_0n_0\). The weighted class 1 proportion is the class 1 mass divided by total weighted mass. This is generally not the raw proportion \(n_1/n\).
If \(W_j=w_1n_{1j}+w_0n_{0j}\) is the weighted mass in child \(j\), and \(W=\sum_j W_j\), the weighted Gini gain is the weighted parent impurity minus the weighted average child impurity:
The weighting in this formula follows from treating each observation as contributing its class weight to the impurity calculation. A child containing more weighted mass contributes more to the post-split criterion. Replacing \(W_j/W\) with the raw fraction of observations while still using weighted child impurities combines two different schemes and can give an incorrect gain.
Why the Weighted Proportion Changes the Decision
A weighted majority prediction chooses class 1 when its weighted mass exceeds class 0's weighted mass. The condition is \(w_1n_1>w_0n_0\). Since \(n_1=np\) and \(n_0=n(1-p)\), this is equivalent to \(w_1p>w_0(1-p)\), and hence to \(p>w_0/(w_0+w_1)\). Equality gives a tie, so the model's tie rule matters at the boundary.
This derivation explains two effects of weighting. First, the weighted proportion \(q\), and therefore impurity, changes. Second, the point at which the weighted majority changes class need not be a raw 50-50 split. Larger \(w_1\) lowers the raw class 1 proportion needed for a class 1 weighted majority.
For class weights inversely proportional to class frequencies, the total weighted mass of each class can be made equal in the training data. That can make rare-class observations more influential in split evaluation. It does not mean every node has equal weighted class masses, and it does not guarantee that the chosen tree will perform better on new data.
Worked Example: Recalculating Split Gain With Class Weights
A hypothetical training node has 10 class 1 and 90 class 0 observations. A candidate split creates a left child with 8 class 1 and 12 class 0 observations, and a right child with 2 class 1 and 78 class 0 observations. Compare the unweighted Gini gain with the gain using \(w_1=9\) and \(w_0=1\).
Method. Calculate the parent and child impurities first under raw counts, then repeat using weighted class masses and weighted child sizes. Keep the two gain calculations separate; their impurity scales reflect different fitting criteria.
Unweighted parent. The raw class 1 proportion is \(10/100=0.1\), so \(G_{\text{parent}}=2(0.1)(0.9)=0.18\).
Unweighted children. The left proportion is \(8/20=0.4\), so \(G_L=2(0.4)(0.6)=0.48\). The right proportion is \(2/80=0.025\), so \(G_R=2(0.025)(0.975)=0.04875\). Their size-weighted impurity is \((20/100)(0.48)+(80/100)(0.04875)=0.096+0.039=0.135\). The unweighted gain is \(0.18-0.135=0.045\).
Weighted parent. The weighted class masses are \(9(10)=90\) and \(1(90)=90\), so \(q=90/180=0.5\) and \(G_{w,\text{parent}}=0.5\).
Weighted left child. Its weighted masses are \(9(8)=72\) and \(1(12)=12\), for total weighted mass \(84\). Thus \(q_L=72/84=6/7\) and \(G_{w,L}=2(6/7)(1/7)=12/49\).
Weighted right child. Its weighted masses are \(9(2)=18\) and \(1(78)=78\), for total weighted mass \(96\). Thus \(q_R=18/96=3/16\) and \(G_{w,R}=2(3/16)(13/16)=39/128\).
Weighted child average and gain. The total weighted mass is \(84+96=180\). Therefore \((84/180)(12/49)+(96/180)(39/128)=(7/15)(12/49)+(8/15)(39/128)=4/35+13/80=0.276785714\ldots\). The weighted gain is \(0.5-0.276785714\ldots=0.223214285\ldots\), or \(0.223214\) to six decimal places.
Check. Across the children, the raw counts are 10 class 1 and 90 class 0, and the weighted masses are 90 and 90, matching the parent. Weighting changes the gain from \(0.045\) to approximately \(0.223214\) because it changes both the impurity proportions and the child weights. The common mistake is to use child count fractions \(20/100\) and \(80/100\) in the weighted calculation; the correct fractions are \(84/180\) and \(96/180\).
Worked Example: A Weighted Majority Prediction at a Node
A hypothetical terminal node contains 12 class 1 and 88 class 0 observations. The fitted tree uses \(w_1=4\) and \(w_0=1\). Find the raw class 1 proportion, the weighted class 1 proportion, and the predicted class under the weighted-majority rule. Then identify the raw-proportion threshold for this rule.
Method. Compare raw proportions with weighted masses. Do not infer the predicted label from the raw majority when the model uses class weights.
The raw class 1 proportion is \(12/100=0.12\), so class 0 is the raw majority. The weighted class 1 mass is \(4(12)=48\); the weighted class 0 mass is \(1(88)=88\). Hence \(q=48/(48+88)=48/136=6/17\approx0.3529\). Since \(48<88\), the weighted-majority prediction is still class 0.
The raw-proportion threshold for predicting class 1 is \(w_0/(w_0+w_1)=1/(1+4)=0.2\). Because the observed raw proportion \(0.12\) is below \(0.2\), the threshold calculation confirms the weighted-mass comparison.
Check. The weighted proportion \(6/17\) is greater than the raw proportion \(0.12\), as expected because class 1 received the larger weight. It is nevertheless below \(0.5\), so the predicted label remains class 0. A common error is to say that any increase in the rare class's weighted proportion must reverse the prediction; the comparison must be made with \(0.5\), or equivalently with the derived raw threshold.
What Class Weights Do—and Do Not Do
Class weights can make a split that separates rare-class observations more influential than it would be under raw counts. They can also change which label is assigned by a weighted-majority rule. The effect depends on the weights, the observations in each candidate child, and the criterion used. A split that appears attractive under weighted Gini need not be best under unweighted Gini, and gains calculated under different weighting schemes should not be compared as though they were the same objective.
Weighting is not a universal correction for imbalance. It may increase sensitivity to the rare class while also increasing false positives. Weighted class proportions are not automatically calibrated estimates of real-world class probabilities: they describe proportions under the weighted criterion. If probabilities are used for decisions, the interpretation of weights and the decision rule both matter. The earlier material on classification thresholds and unequal error costs distinguishes a probability cutoff from a tree's split criterion; changing class weights affects the criterion and may affect node predictions, but does not by itself establish that a chosen threshold is appropriate for every application.
When assessing a tree, report what the metric measures. Overall accuracy can obscure rare-class errors, while class-specific summaries make those errors visible. For split calculations, state whether the counts are raw or weighted, use the matching child weights, and identify the tie rule if weighted masses are equal. These checks prevent a high majority-class proportion from being mistaken for evidence that the model is useful for the less common class.
Check Your Understanding
For each problem, use the stated rule and keep raw counts distinct from weighted quantities.
- A node has 7 class 1 and 93 class 0 observations. What label does an unweighted majority rule predict, what is its accuracy on this node, and how many class 1 observations does it miss? (Answer: class 0; 93%; 7 missed.)
- A node has 15 class 1 and 45 class 0 observations, with \(w_1=3\) and \(w_0=1\). Find its weighted class 1 proportion and state whether the weighted masses tie. (Answer: \(45/(45+45)=0.5000\); they tie, so the label depends on the stated tie rule.)
- A parent has 10 class 1 and 90 class 0 observations with \(w_1=9,w_0=1\). A split gives children \((8,12)\) and \((2,78)\), where each pair lists class 1 then class 0 counts. What is the weighted Gini gain? (Answer: approximately \(0.223214\).)
- For \(w_1=5\) and \(w_0=1\), derive the raw class 1 proportion at which weighted masses tie. If a node's raw class 1 proportion is \(0.14\), what is its weighted-majority prediction? (Answer: tie threshold \(1/6\approx0.1667\); predict class 0.)
- A training set has 12 class 1 and 108 class 0 observations. Suppose inverse-frequency weights are used, so each class receives the same total weighted mass. What ratio \(w_1/w_0\) achieves this, and what is the weighted class 1 proportion in the parent? (Answer: \(w_1/w_0=108/12=9\); the weighted class 1 proportion is \(0.5000\).)