Why Multiclass Splits Need Careful Accounting
A classification tree may need to distinguish among three or more outcomes, such as several claim categories or several types of equipment failure. A useful split separates observations so that the resulting child nodes are more concentrated in their class membership than the parent node. To measure that concentration, calculate an impurity from the class proportions in each node, then average the child impurities using the numbers of observations sent to them.
The key practical points are straightforward but easy to mishandle: include every class in the calculation, use proportions rather than raw counts inside the impurity formula, and weight child impurities by child size. The most common predicted class in a terminal node is its class prediction; a node can have a tie, in which case there is no unique majority class.
Both measures are zero when all observations belong to one class. They are larger when the class proportions are more evenly distributed. Their maximum value depends on the number of classes: for \(K\) equally frequent classes, \(G=1-1/K\) and \(H=\log_2(K)\). Therefore, when interpreting an impurity, keep the number of possible classes in view.
Calculating Impurity and Split Gain
Suppose a candidate split sends the parent node’s \(n\) observations into \(J\) nonempty child nodes. Let child \(j\) have size \(n_j\), and let \(p_{jk}\) be the proportion of its observations in class \(k\). The child’s Gini impurity and entropy are calculated from its own class proportions. The split’s weighted impurity is the average across children, with weights equal to the fraction of the parent observations in each child.
The corresponding impurity reductions are the parent impurity minus the weighted child impurity:
For a given impurity measure, a larger reduction means a greater decrease in that measure’s weighted impurity. The units differ: Gini reduction is on the Gini scale, while entropy reduction is in bits. The reductions should be calculated using the same measure throughout a comparison.
Worked Example: Calculating Both Impurities for a Three-Class Split
A hypothetical claim-review node contains 12 observations: 6 routine claims, 3 claims requiring further documentation, and 3 claims referred for specialist review. A proposed split produces two child nodes, each with 6 observations. The first child contains counts \((4,1,1)\) across those three classes; the second contains \((2,2,2)\). Calculate the parent and child impurities, the weighted reductions, and the predicted class in each child.
Method. Convert each count vector to proportions, calculate each node’s Gini impurity and entropy, and weight the two child results by their sizes. Each child contains 6 of the parent’s 12 observations, so each weight is \(6/12=1/2\).
Parent. The proportions are \((6/12,3/12,3/12)=(1/2,1/4,1/4)\). Thus
The parent entropy is
First child. Its proportions are \((4/6,1/6,1/6)=(2/3,1/6,1/6)\). Therefore
For entropy, the three terms are approximately \(0.3900\), \(0.4308\), and \(0.4308\) bits:
Second child. Its proportions are \((1/3,1/3,1/3)\), so
Weighted results and check. Since the weights are both \(1/2\),
so \(\Delta G=5/8-7/12=15/24-14/24=1/24\approx0.0417\). For entropy,
The first child predicts routine claims because that class has the largest count, 4. All three classes tie at 2 in the second child, so it has no unique majority-class prediction; a tree implementation would need a tie-breaking rule. Both calculated reductions are positive, as expected for this partition.
Common mistake avoided. The first child’s entropy is not \(0.9183+0.4308+0.4308\). The \(2/3\) term is \(-(2/3)\log_2(2/3)\approx0.3900\), not \(0.9183\). Using the correct term gives \(H_1\approx1.2516\) bits.
Why Weighted Impurity Cannot Increase
For a partition of a parent node, both Gini impurity and entropy are concave functions of the class-proportion vector. This property explains why the weighted child impurity cannot exceed the parent impurity, apart from numerical rounding. The following derivation shows the result directly for Gini; entropy follows from concavity.
Let \(p_k\) be the parent proportion in class \(k\), and \(p_{jk}\) the corresponding proportion in child \(j\). Because the children partition the parent observations, \(p_k=\sum_jw_jp_{jk}\), where \(w_j=n_j/n\) and \(\sum_jw_j=1\).
Using the vector notation \(p=(p_1,\ldots,p_K)\) and \(p_j=(p_{j1},\ldots,p_{jK})\), the definition gives \(\Delta G=\sum_jw_j\lVert p_j\rVert^2-\lVert p\rVert^2\).
Expand the weighted squared distances and use \(\sum_jw_jp_j=p\): \(\sum_jw_j\lVert p_j-p\rVert^2=\sum_jw_j\lVert p_j\rVert^2-2\lVert p\rVert^2+\lVert p\rVert^2=\Delta G\). Every term on the left is nonnegative, so \(\Delta G\geq0\).
For entropy, the function \(h(x)=-x\log_2(x)\) is concave on \(0<x\leq1\), since \(h''(x)=-1/(x\ln 2)<0\). Summing this function over the class proportions makes \(H\) concave as well. Because the parent proportions are the weighted averages of the child proportions, concavity gives \(H_{\mathrm{parent}}\geq\sum_jw_jH_j\), and hence \(\Delta H\geq0\). A split may leave a reduction at zero; for example, that happens when every child has exactly the parent’s class proportions.
Comparing Candidate Splits
When candidate splits have different child sizes, the size weights are essential. The next example uses Gini reduction to compare two possible partitions of the same three-class parent. It also illustrates that a larger reduction is a comparison within the chosen impurity measure, not a statement about the eventual performance of a fitted tree on new observations.
Worked Example: Choosing Between Two Multiclass Partitions
A hypothetical device-monitoring node contains 10 observations in three outcome classes, with counts \((5,3,2)\). Candidate A creates children with counts \((4,1,1)\) and \((1,2,1)\). Candidate B creates children with counts \((3,2,1)\) and \((2,1,1)\). Calculate each weighted Gini impurity and reduction, then identify the preferred candidate by Gini reduction.
Parent. The parent Gini impurity is
Candidate A. The child sizes are 6 and 4, so the weights are \(6/10=0.6\) and \(4/10=0.4\). For the first child, the proportions are \((4/6,1/6,1/6)\), giving \(G_{A1}=1-(16+1+1)/36=1/2\). For the second, the proportions are \((1/4,2/4,1/4)\), giving \(G_{A2}=1-(1+4+1)/16=5/8=0.625\). Thus
Candidate B. Its children also have sizes 6 and 4. The first child has proportions \((1/2,1/3,1/6)\), so \(G_{B1}=1-(1/4+1/9+1/36)=1-14/36=11/18\). The second has proportions \((1/2,1/4,1/4)\), so \(G_{B2}=1-(1/4+1/16+1/16)=5/8\). Therefore
and \(\Delta G_B=0.62-0.6167\approx0.0033\), or exactly \(31/50-37/60=1/300\). Candidate A is preferred by Gini reduction: \(0.07\) exceeds \(1/300\approx0.0033\). Its two child predictions are the first class for \((4,1,1)\) and the second class for \((1,2,1)\).
Common mistake avoided. Averaging Candidate A’s child impurities equally would give \((0.5+0.625)/2=0.5625\), and an incorrect reduction of \(0.0575\). The children are not the same size, so the correct calculation is \(0.6(0.5)+0.4(0.625)=0.55\).
Entropy Gain and Leaf Predictions
Entropy uses the same child-size weights as Gini, but its node calculation uses logarithms. At a leaf, the predicted class is the class with the largest count in that node. The class proportions can also describe the observed class mix, but the predicted label is not itself an impurity measure.
Worked Example: Entropy Gain and Multiclass Leaf Labels
A hypothetical quality-control node contains 8 observations from three defect categories, with counts \((4,2,2)\). A split creates two children of size 4, with counts \((3,1,0)\) and \((1,1,2)\), respectively. Calculate the entropy reduction and Gini reduction, and state the predicted class in each child.
Parent entropy. The parent proportions are \((1/2,1/4,1/4)\), so
First child. Its proportions are \((3/4,1/4,0)\). The zero-proportion term contributes zero, so
Second child. Its proportions are \((1/4,1/4,1/2)\), giving
Each child has weight \(4/8=1/2\). Thus \(H_{\mathrm{split}}=(0.8113+1.5000)/2=1.1556\) bits, and the entropy reduction is \(1.5000-1.1556=0.3444\) bits, rounded to four decimal places.
As a check using Gini, the parent impurity is \(1-(1/4+1/16+1/16)=0.625\). The first child has \(G_1=1-(9/16+1/16)=0.375\), and the second has \(G_2=1-(1/16+1/16+4/16)=0.625\). The weighted child Gini is \((0.375+0.625)/2=0.5\), so the Gini reduction is \(0.625-0.5=0.125\). The first child predicts the first class (count 3); the second predicts the third class (count 2). Both impurity reductions are nonnegative, which checks the direction of the calculations.
Common Calculation Checks
Before accepting a multiclass split calculation, check that the child counts add back to the parent count class by class, the proportions in each node sum to 1, and the child weights sum to 1. Then check that each reduction equals parent impurity minus weighted child impurity. These checks catch many arithmetic and setup errors.
- Do not omit a class. A class with zero observations contributes \(0\) to Gini’s squared-proportion sum and contributes \(0\) to entropy under the zero-term convention. It should not be treated as an undefined logarithm.
- Do not substitute counts for proportions. For Gini, squaring raw counts without dividing by the squared node size does not give an impurity.
- Do not average child impurities without size weights. Equal weights are correct only when the child nodes have equal sizes.
- Do not use a child’s most common class as its impurity. The predicted label uses the largest count; Gini and entropy use all class proportions.
- Check the direction of the reduction. A negative result for a partition of a node indicates an arithmetic or weighting error, aside from small rounding discrepancies.
For an aligned-with-the-SOA-Exam-SRM-syllabus treatment of classification trees, the central multiclass calculation is the same regardless of the application: count every class in each node, compute its proportions, apply the selected impurity formula, and compare candidates using size-weighted child values. The leaf label is then read from the largest class count, with ties handled explicitly rather than silently resolved.
Check Your Understanding
For each problem, calculate from the class counts and retain enough precision to distinguish exact values from rounded decimals.
- A node has class counts \((3,2,1)\). Calculate its Gini impurity. (Answer: \(11/18\approx0.6111\).)
- A node has class proportions \((1/2,1/4,1/4)\). Calculate its entropy in bits. (Answer: \(1.5000\) bits.)
- A parent has counts \((4,3,1)\), and a split gives children \((3,1,0)\) and \((1,2,1)\). Calculate the parent Gini, weighted child Gini, and reduction; give the predicted class in each child. (Answer: parent \(19/32=0.59375\); weighted child \(0.5000\); reduction \(3/32=0.09375\); first child predicts class 1 and second predicts class 2.)
- A parent has counts \((3,2,1)\). A split gives children \((2,1,0)\) and \((1,1,1)\). Calculate the parent entropy, weighted child entropy, and entropy reduction. (Answer: parent \(\approx1.4591\) bits; weighted child \(\approx1.2516\) bits; reduction \(\approx0.2075\) bits.)
- A parent has counts \((5,4,3)\), and a split gives equal-sized children \((4,1,1)\) and \((1,3,2)\). Calculate the parent and weighted child Gini impurities, the reduction, and each child prediction. (Answer: parent \(47/72\approx0.6528\); weighted child \(5/9\approx0.5556\); reduction \(7/72\approx0.0972\); first child predicts class 1 and second predicts class 2.)