Why Compare Impurity Measures?
A classification tree divides observations into groups intended to have similar class labels. To compare candidate divisions, a tree can measure how mixed the classes are in each resulting node. Gini impurity, entropy, and classification error all describe this mixing, but they emphasize it differently. In particular, classification error can overlook changes in class proportions that Gini and entropy register.
The distinction matters when interpreting why a split is preferred. A measure that changes little is not necessarily evidence that the class proportions barely changed: it may simply be a less sensitive measure of those changes. The comparisons here are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
For entropy, a term with \(p_k=0\) is defined to be zero, consistent with the limit of \(-p_k\log_2(p_k)\) as \(p_k\) approaches zero. Gini impurity and classification error range from zero to less than one; entropy is also zero for a pure node and is largest when the class proportions are equal. Their numerical scales differ, so compare changes within a criterion rather than treating a Gini value and an entropy value as directly comparable.
Each measure is an empirical description of class composition in a node. For a split, the resulting children form a disjoint partition of the parent observations. The correct combined child score weights each child's impurity by its share of the parent.
Scoring a Candidate Split
Let a parent node contain \(N\) observations. A candidate split creates \(J\) children; child \(j\) has \(N_j\) observations, with \(\sum_{j=1}^{J}N_j=N\). Write \(w_j=N_j/N\), and let \(I_j\) denote the impurity of child \(j\) under whichever criterion is being used. The weighted child impurity and the impurity reduction are
A larger positive reduction indicates a greater decrease in the chosen measure of mixing. The weights matter: a child containing 90% of the observations should contribute more to the combined score than one containing 10%.
Each parenthesized expression is a weighted variance: it is the weighted mean of squared child proportions minus the square of their weighted mean. It is nonnegative, which explains why the weighted Gini reduction cannot be negative for a genuine partition of the parent. Entropy and classification error are also concave functions of the class-proportion vector, so their weighted child impurity cannot exceed the parent impurity. This comparison assumes the same observations and class definitions are used throughout.
Worked Example: A Change the Error Measure Does Not See
A node contains 10 hypothetical claims classified into three mutually exclusive categories. In one possible composition, the category counts are \((6,2,2)\); in another, they are \((6,3,1)\). Compare the three node impurity measures. Method: convert counts to proportions and evaluate each definition. This comparison avoids the mistake of assuming that an unchanged majority-class prediction means unchanged impurity.
For counts \((6,2,2)\), the proportions are \((0.6,0.2,0.2)\). Thus
Using base-2 logarithms,
The largest proportion is \(0.6\), so \(E_1=1-0.6=0.4\).
For counts \((6,3,1)\), the proportions are \((0.6,0.3,0.1)\). Therefore,
and \(E_2=1-0.6=0.4\). The majority class is unchanged, so the classification error is identical. Gini impurity decreases by \(0.02\), and entropy decreases by approximately \(0.075489\). Each measure captures a different aspect of class composition; here, the two smaller class proportions changed, but the largest proportion did not.
Why Classification Error Is Less Sensitive
Gini impurity uses every class proportion through the sum of their squares. Entropy uses every proportion through a logarithmic term. Classification error, in contrast, uses only the largest proportion. If the same class remains the most common, changes to the other proportions may leave \(E=1-\max_k p_k\) exactly unchanged.
This does not mean classification error always changes less in every numerical comparison. Its key limitation is that it is flat over regions where the identity of the majority class does not change: there, small redistributions among the other classes produce no change in the measure. Gini and entropy usually respond to those redistributions. This difference helps explain why Gini and entropy are often used to assess candidate splits, while classification error gives a comparatively coarse summary of the dominant-class prediction.
Worked Example: Two Splits With the Same Classification Error
A hypothetical parent node has 20 observations: 16 in class A and 4 in class B. Candidate split A produces two children with counts \((8,2)\) and \((8,2)\). Candidate split B produces children with counts \((5,0)\) and \((11,4)\). Compare their weighted child impurities. Method: compute each child score, weight it by child size, and compare reductions from the common parent. This avoids mistaking an unweighted average for a split score.
The parent proportions are \((0.8,0.2)\). Its Gini impurity is \(1-(0.8^2+0.2^2)=0.32\), its entropy is \(-0.8\log_2(0.8)-0.2\log_2(0.2)\approx0.721928\), and its classification error is \(1-0.8=0.2\).
For candidate split A, both children have proportions \((0.8,0.2)\). Their weighted Gini impurity is \[ (10/20)(0.32)+(10/20)(0.32)=0.32, \] so its Gini reduction is \(0.32-0.32=0\). Its weighted entropy is \(0.721928\), giving entropy reduction \(0\). Its weighted classification error is \((10/20)(0.2)+(10/20)(0.2)=0.2\), giving error reduction \(0\).
For candidate split B, the first child is pure, with proportions \((1,0)\), so its Gini impurity, entropy, and error are all zero. The second child has proportions \((11/15,4/15)\). Its Gini impurity is
Its entropy is \[ -\frac{11}{15}\log_2(11/15)-\frac{4}{15}\log_2(4/15)\approx0.836640. \] The second child's weight is \(15/20=0.75\), so the weighted child Gini impurity is \(0.75(0.391111)\approx0.293333\), and its reduction is \(0.32-0.293333\approx0.026667\). The weighted child entropy is \(0.75(0.836640)\approx0.627480\), and its reduction is \(0.721928-0.627480\approx0.094448\). The second child's error is \(1-11/15=4/15\); its weighted contribution is \(0.75(4/15)=0.2\). Thus the weighted child error remains \(0.2\), just as for split A.
Gini and entropy distinguish the splits, assigning a positive reduction to split B and none to split A. Classification error ties them. This calculation compares split scores only; it does not specify a rule for when to stop growing a tree.
A Multiclass Split Comparison
The measures can also be compared directly when several classes are present. The next example computes the parent impurity and the weighted child scores for two candidate splits, using the same 12 observations in each case.
Worked Example: Comparing Two Three-Class Splits
A hypothetical parent node has class counts \((6,4,2)\). Candidate X creates children \((4,1,1)\) and \((2,3,1)\). Candidate Y creates children \((5,1,0)\) and \((1,3,2)\). Each child has size 6. Calculate each criterion's weighted child impurity and reduction. Method: apply the three definitions to each proportion vector, average the equally sized children, then subtract from the parent score. This avoids comparing child impurities without their proper weights.
The parent proportions are \((1/2,1/3,1/6)\). Its Gini impurity is \(1-(1/4+1/9+1/36)=11/18\approx0.611111\); its entropy is \(0.5+0.528321+0.430827\approx1.459148\); and its classification error is \(1-1/2=0.5\).
For candidate X, the child proportions are \((4/6,1/6,1/6)\) and \((2/6,3/6,1/6)\). The first child's Gini impurity is \(1-(16+1+1)/36=0.5\); the second's is \(1-(4+9+1)/36=22/36\approx0.611111\). Their weighted child Gini impurity is \((0.5+0.611111)/2\approx0.555556\), giving reduction \(0.611111-0.555556\approx0.055556\).
The first child's entropy is approximately \(1.251629\), and the second's is approximately \(1.459148\). Thus weighted child entropy is \((1.251629+1.459148)/2\approx1.355389\), giving reduction \(1.459148-1.355389\approx0.103759\). The children have errors \(1-4/6=2/6\) and \(1-3/6=3/6\), so weighted child error is \((2/6+3/6)/2=5/12\approx0.416667\), giving reduction \(0.5-0.416667\approx0.083333\).
For candidate Y, the child proportions are \((5/6,1/6,0)\) and \((1/6,3/6,2/6)\). Their Gini impurities are \(1-(25+1)/36=10/36\approx0.277778\) and \(1-(1+9+4)/36=22/36\approx0.611111\). The weighted child Gini impurity is \((0.277778+0.611111)/2\approx0.444444\), giving reduction \(0.611111-0.444444\approx0.166667\).
The first child's entropy is approximately \(0.650022\), and the second's is approximately \(1.459148\). Weighted child entropy is \((0.650022+1.459148)/2\approx1.054585\), giving reduction \(1.459148-1.054585\approx0.404563\). The child errors are \(1/6\) and \(3/6\), so weighted child error is \((1/6+3/6)/2=1/3\approx0.333333\), giving reduction \(0.5-0.333333\approx0.166667\). All three criteria favor candidate Y in this example, although the sizes of their reductions differ.
Common Mistake: Averaging Children Equally
A split score is a size-weighted average, not an unweighted average of child scores unless the children have equal sizes. In the previous example, candidate B in the two-class comparison produced a pure child of size 5 and a child of size 15 with classification error \(4/15\). Incorrectly averaging the two child errors equally gives \[ (0+4/15)/2=2/15\approx0.1333. \] The correct weighted error is \[ (5/20)(0)+(15/20)(4/15)=0.2. \] Candidate A also has weighted error \(0.2\), so the correct conclusion is a tie under classification error. The unweighted calculation falsely suggests a smaller error for candidate B.
When comparing criteria, also keep the interpretation precise: a lower weighted child impurity means a larger reduction only when comparing splits under the same criterion and parent. A Gini value should not be directly compared numerically with an entropy value, because they use different scales.
Check Your Understanding
For each problem, compute the requested quantity from the stated class proportions or counts.
- A node has three class proportions \((0.5,0.3,0.2)\). Find its Gini impurity, entropy in bits, and classification error. (Answer: Gini \(=0.62\); entropy \(\approx1.4855\) bits; classification error \(=0.5\).)
- A node has 20 observations, 14 in class A and 6 in class B. A split produces children \((7,1)\) and \((7,5)\), with counts listed as (A, B). Find the weighted child classification error. (Answer: \(0.30\).)
- A parent has counts \((12,8)\) in classes A and B. Candidate P produces children \((9,1)\) and \((3,7)\); candidate Q produces \((10,0)\) and \((2,8)\). Compare their weighted child classification errors and Gini impurities. (Answer: P has error \(0.20\) and Gini \(0.30\); Q has error \(0.10\) and Gini \(0.16\), so Q has the lower score under both criteria.)
- A parent has counts \((8,4,4)\). Split R gives children \((4,2,2)\) and \((4,2,2)\); split S gives children \((8,0,0)\) and \((0,4,4)\). For each split, calculate the weighted child error and say whether the splits tie under that criterion. (Answer: R has weighted error \(0.50\); S has weighted error \(0.25\); they do not tie.)