Selecting Splits With Gini Impurity

Why Split Choice Requires a Weighted Comparison

A classification tree divides observations into groups so that the class labels within each group are more homogeneous. At a node, several candidate partitions may be available. Choosing among them requires comparing the impurity of the parent node with the impurity that remains across all of a candidate’s child nodes. A small, pure child is not enough to make a split good: the other child may contain most observations and remain highly mixed.

This tutorial focuses on binary classification, with two possible labels. For example, a hypothetical insurer might classify whether a claim is referred for review or not. The calculations use the observations in the node being split; no claim is made about an actual insurer or dataset. The candidate partitions are compared using Gini impurity and the reduction in Gini impurity.

Definition: If a node contains class proportions \(p_1\) and \(p_0\), where \(p_1+p_0=1\), its Gini impurity is \(G=1-p_1^2-p_0^2\). It is zero when every observation belongs to one class and is largest at \(0.5\) when the two classes are equally represented.

For a candidate split, let the children be indexed by \(j\), let \(n_j\) be the number of observations in child \(j\), and let \(n\) be the number in the parent. The weight of child \(j\) is \(n_j/n\). Its Gini impurity is calculated from the class proportions within that child, not from the parent proportions.

Key result: The weighted post-split Gini impurity is the sum of each child’s Gini impurity multiplied by that child’s share of the parent observations. The Gini reduction is the parent impurity minus this weighted post-split impurity. Among the candidate splits being compared, choose the split with the largest reduction.

G_{\text{post}}=\sum_j \frac{n_j}{n}G_j \qquad\text{and}\qquad \Delta G=G_{\text{parent}}-G_{\text{post}}.

In the usual unweighted calculation, every observation contributes equally, so child size determines its weight. The children of a split must partition the parent observations: each observation goes into exactly one child. If two candidates partition the same parent, they have the same parent impurity, so minimizing weighted post-split impurity is equivalent to maximizing Gini reduction. A reduction of zero means the split does not improve this impurity measure; a larger positive reduction indicates a greater decrease.

Why the Weighted Formula Works

Gini impurity has a probability interpretation. Imagine drawing two labels independently from the class proportions in a node. The probability that the labels differ is \(1-(p_1^2+p_0^2)\): the probability that both are class 1 is \(p_1^2\), and the probability that both are class 0 is \(p_0^2\). Thus Gini impurity measures the probability of different labels in this comparison.

After a split, an observation belongs to child \(j\) with frequency \(n_j/n\) in the training node. The impurity expected across children is therefore the sum of each child’s impurity weighted by that frequency. Subtracting that expected impurity from the parent’s impurity gives the decrease attributable to the partition:

\Delta G =G_{\text{parent}}-\sum_j\frac{n_j}{n}G_j.

This calculation does not treat a tiny child as though it represented half the data. It also does not say that the selected split must improve predictive performance on new observations; it measures impurity reduction in the observations at the current node. The calculation here assumes ordinary, equally weighted observations and a binary response.

Comparing Candidate Partitions

Worked Example: Choosing Between Two Review-Flag Splits

A hypothetical training node contains 10 claims: 6 are referred for review and 4 are not. A candidate split A creates children with counts (4 referred, 1 not referred) and (2 referred, 3 not referred). Candidate split B creates children with counts (2 referred, 0 not referred) and (4 referred, 4 not referred). Calculate each weighted post-split impurity and select the better split by Gini reduction. Use proportions without rounding until the final comparison.

Method. Calculate the parent Gini impurity, then each child’s Gini impurity, weight by child size, and compare the reductions. This avoids choosing a split solely because one child is pure.

Parent. The class proportions are \(6/10=0.6\) and \(4/10=0.4\). Therefore,

G_{\text{parent}}=1-(0.6)^2-(0.4)^2 =1-0.36-0.16=0.48.

Candidate A. Each child has 5 observations. In the first child, the proportions are \(4/5\) and \(1/5\), so its impurity is \(1-(4/5)^2-(1/5)^2=1-16/25-1/25=8/25=0.32\). In the second child, the proportions are \(2/5\) and \(3/5\), giving \(1-4/25-9/25=12/25=0.48\). The weighted impurity is

G_{\text{post,A}} =\frac{5}{10}(0.32)+\frac{5}{10}(0.48) =0.16+0.24=0.40.

Its reduction is \(0.48-0.40=0.08\).

Candidate B. The first child is pure, so its Gini impurity is \(1-1^2-0^2=0\). The second child has equal class proportions \(4/8=1/2\), so its impurity is \(1-(1/2)^2-(1/2)^2=0.50\). The weighted impurity and reduction are

G_{\text{post,B}} =\frac{2}{10}(0)+\frac{8}{10}(0.50)=0.40, \qquad \Delta G_B=0.48-0.40=0.08.

The candidates tie: each has weighted post-split impurity \(0.40\) and Gini reduction \(0.08\). Candidate B has a pure child, but that child contains only 2 of the 10 claims; its other child is evenly mixed. The weighted calculation correctly accounts for both. Check: For each split, the child counts add to 10, and parent impurity minus post-split impurity equals the stated reduction. Rounding is not needed.

Child Size Can Change the Ranking

Worked Example: Avoiding an Unweighted Average

A second hypothetical node contains 40 observations, evenly divided between two labels: 20 in class 1 and 20 in class 0. Candidate A divides the data into two groups of 20, with class counts (15, 5) and (5, 15). Candidate B creates a pure child with counts (2, 0), leaving (18, 20) in the other child. Calculate the weighted post-split impurity and reduction for each candidate. Then compare with the incorrect approach of giving the two children equal weights.

Parent impurity. The proportions are \(1/2\) and \(1/2\), so \(G_{\text{parent}}=1-(1/2)^2-(1/2)^2=0.50\).

Candidate A. The first child has proportions \(15/20=0.75\) and \(5/20=0.25\), giving impurity \(1-0.75^2-0.25^2=1-0.5625-0.0625=0.375\). The second child reverses those proportions and also has impurity \(0.375\). Both weights are \(20/40=0.5\). Thus,

G_{\text{post,A}}=0.5(0.375)+0.5(0.375)=0.375, \qquad \Delta G_A=0.50-0.375=0.125.

Candidate B. The child with counts (2, 0) has impurity zero. The other child has 38 observations and proportions \(18/38\) and \(20/38\). Its impurity is

1-\left(\frac{18}{38}\right)^2-\left(\frac{20}{38}\right)^2 =1-\frac{324+400}{1444} =\frac{720}{1444} =\frac{180}{361}.

The child weights are \(2/40\) and \(38/40\). Therefore,

G_{\text{post,B}} =\frac{2}{40}(0)+\frac{38}{40}\left(\frac{180}{361}\right) =\frac{171}{361} \approx 0.4737, \qquad \Delta G_B=0.50-\frac{171}{361} =\frac{19}{722} \approx 0.0263.

Candidate A is better because its reduction, \(0.125\), exceeds \(0.0263\). If the two children of candidate B were incorrectly given equal weights, its post-split value would be \((0+180/361)/2=90/361\approx0.2493\), which would misleadingly make B seem better than A’s \(0.375\). This is the common mistake: equal weighting gives a child containing 2 observations the same influence as one containing 38. Check: The weighted impurity of B is close to the large child’s impurity because that child contains \(38/40\) of the observations. Final values are rounded to four decimal places.

Use the Reduction, Not Just the Child Counts

Worked Example: A Small Pure Child Versus a More Balanced Split

A third hypothetical node has 20 observations: 12 in class 1 and 8 in class 0. Candidate A produces children (6, 2) and (6, 6). Candidate B produces children (3, 0) and (9, 8). Calculate the weighted post-split impurity and Gini reduction for each; state which split is preferred by this criterion.

Parent impurity. The proportions are \(12/20=0.6\) and \(8/20=0.4\), so \(G_{\text{parent}}=1-0.36-0.16=0.48\).

Candidate A. The first child has 8 observations and proportions \(6/8=0.75\) and \(2/8=0.25\), so its impurity is \(1-0.5625-0.0625=0.375\). The second child has proportions \(6/12=0.5\) and \(6/12=0.5\), so its impurity is \(0.50\). The weights are \(8/20=0.4\) and \(12/20=0.6\). Hence,

G_{\text{post,A}}=0.4(0.375)+0.6(0.50) =0.15+0.30=0.45, \qquad \Delta G_A=0.48-0.45=0.03.

Candidate B. The (3, 0) child is pure and has impurity zero. The other child has 17 observations, with proportions \(9/17\) and \(8/17\). Its impurity is

1-\left(\frac{9}{17}\right)^2-\left(\frac{8}{17}\right)^2 =1-\frac{81+64}{289} =\frac{144}{289}.

The child weights are \(3/20\) and \(17/20\). The weighted post-split impurity is

G_{\text{post,B}} =\frac{3}{20}(0)+\frac{17}{20}\left(\frac{144}{289}\right) =\frac{144}{340} =\frac{36}{85} \approx0.4235.

Thus \(\Delta G_B=0.48-36/85=24/425\approx0.0565\), which is greater than candidate A’s \(0.03\). Choose B by Gini reduction, even though its large child remains nearly as mixed as the parent: the small pure child makes enough of a weighted contribution to produce the larger reduction. Check: The child sizes \(3\) and \(17\) sum to \(20\), and \(0.48-0.4235\approx0.0565\). Final decimal values are rounded to four places.

Common Exam Traps

Taking an unweighted average. Use \(n_j/n\) for each child, not an equal share for every child, unless the child sizes actually are equal.
Looking only at the purest child. A pure child can be very small while the large child remains highly mixed. Calculate the full weighted post-split impurity.
Comparing reductions without a common parent. For candidate splits of the same node, parent impurity is fixed, so the largest reduction is the smallest weighted post-split impurity. If the parent is the same, those rankings agree.
Using parent proportions inside a child. Recalculate class proportions separately for every child before finding its Gini impurity.
Confusing a training-node criterion with a guarantee about future data. Gini reduction describes the observed node partition. It does not by itself establish how well a split will perform on new observations.

Key takeaway: For each candidate split, calculate every child’s Gini impurity, weight it by the fraction of parent observations in that child, and subtract the result from the parent impurity. Select the candidate with the largest Gini reduction; never let a small child count as much as a large one.

Check Your Understanding

For each question, use the binary-class Gini formula and child-size weights. Round final decimal values to four places when needed.

A node has 5 class-1 and 3 class-0 observations. A split creates children (3, 1) and (2, 2). Find the parent impurity, weighted post-split impurity, and reduction. (Answer: parent \(0.4688\); post-split \(0.4375\); reduction \(0.0313\).)
A node has counts (6, 4). Candidate A creates children (4, 0) and (2, 4); candidate B creates (3, 2) and (3, 2). Calculate both reductions and select the better candidate. (Answer: A has post-split impurity \(0.2667\) and reduction \(0.2133\); B has reduction \(0\); choose A.)
A node has counts (9, 9). A split creates children (7, 2) and (2, 5). Find the weighted post-split impurity and reduction. (Answer: post-split \(188/567\approx0.3316\); reduction \(191/1134\approx0.1684\).)
A node has counts (8, 4). A split creates children (4, 0) and (4, 4). Find the parent impurity, weighted post-split impurity, and reduction. (Answer: parent \(4/9\approx0.4444\); post-split \(1/3\approx0.3333\); reduction \(1/9\approx0.1111\).)
A node has counts (10, 10). Candidate A creates children (8, 2) and (2, 8); candidate B creates (1, 0) and (9, 10). Calculate both weighted post-split impurities and state which candidate is preferred. (Answer: A has post-split impurity \(0.3200\) and reduction \(0.1800\); B has post-split impurity \(9/19\approx0.4737\) and reduction \(1/38\approx0.0263\); choose A.)

What You'll Learn