A Hand Calculation of Entropy Gain

Why Calculate Entropy Gain by Hand?

A classification tree chooses a split by asking how much that split reduces uncertainty about the class label. For a small node, the calculation is manageable by hand, but several details matter: use the class proportions in each node, weight each child by its share of the parent observations, and compare gains only for valid partitions of the same parent. This tutorial works through those calculations for binary labels.

The label names are arbitrary. In the examples, “positive” and “negative” identify the two possible classes; they do not mean that one outcome is desirable. Each example treats the observed class counts as the empirical proportions used to calculate node entropy.

Definition: If a node contains $n_+$ positive and $n_-$ negative observations, with $n=n_++n_-$, its binary entropy is $$ H=-p_+\log_2(p_+)-p_-\log_2(p_-), \qquad p_+=\frac{n_+}{n},\quad p_-=\frac{n_-}{n}. $$ A term with probability zero is defined as zero, so a pure node has entropy zero. Entropy is measured in bits; a node with equal class proportions has entropy one bit.

A candidate split partitions the parent observations among child nodes. If child $j$ contains $n_j$ observations and has entropy $H_j$, the weighted child entropy is $\sum_j (n_j/n)H_j$. The information gain is the parent entropy minus this weighted average. A larger gain means that the split produces a greater reduction in class uncertainty, according to this criterion.

From Counts to Weighted Information Gain

The calculation follows the same sequence every time: first calculate the parent proportions and entropy; then calculate each child’s proportions and entropy; next weight each child entropy by its size relative to the parent; finally subtract the weighted child entropy from the parent entropy. For a binary node, it is enough to calculate the positive proportion, since the negative proportion is its complement.

\operatorname{Gain} = H_{\mathrm{parent}} - \sum_{j=1}^{J}\frac{n_j}{n}H_j, \qquad \sum_{j=1}^{J} n_j=n.

The child-count condition is essential: the children must contain every parent observation exactly once. A candidate split that loses observations, duplicates them, or changes the parent class totals is not a valid partition and cannot be compared as a split of that parent.

Calculate the parent entropy.
Use the parent’s positive and negative counts to find its class proportions and apply the binary entropy formula.

Calculate each child entropy.
Use the class counts within that child, not the parent proportions.

Weight and add.
Multiply each child entropy by the child’s fraction of the parent observations, then add the contributions.

Subtract and compare.
Subtract the weighted child entropy from the parent entropy. The larger valid gain is preferred by the entropy criterion.

Worked Calculations

Worked Example: Comparing Two Splits of a Balanced Parent

A hypothetical screening dataset has six records at a parent node: three positive and three negative. Candidate A produces children with counts $(2+,1-)$ and $(1+,2-)$. Candidate B produces children with counts $(1+,1-)$ and $(2+,2-)$. Calculate both gains and identify the preferred split.

Method. Calculate the parent entropy, then the size-weighted entropy for each candidate. Check that each candidate’s child counts sum to the parent counts.

Parent. The positive and negative proportions are both $3/6=1/2$. Therefore,

H_{\mathrm{parent}} = -\frac12\log_2\left(\frac12\right) -\frac12\log_2\left(\frac12\right) =0.500000+0.500000=1.000000\text{ bit}.

Candidate A. Each child has three observations. The first has proportions $2/3$ and $1/3$, and the second has proportions $1/3$ and $2/3$. Both have the same entropy:

H_A = -\frac23\log_2\left(\frac23\right) -\frac13\log_2\left(\frac13\right) =0.389975+0.528321 =0.918296\text{ bits}.

The weighted child entropy is $(3/6)(0.918296)+(3/6)(0.918296)=0.918296$ bits. Thus Candidate A’s gain is $1.000000-0.918296=0.081704$ bits, or 0.0817 bits to four decimal places. Its children sum to three positive and three negative records, as required.

Candidate B. Both children are balanced: the first has one positive and one negative, and the second has two of each. Each child entropy is one bit. The weighted child entropy is $(2/6)(1)+(4/6)(1)=1.000000$ bit, so Candidate B’s gain is $1.000000-1.000000=0.000000$ bits, or 0.0000 bits.

Conclusion and check. Candidate A is preferred because $0.0817>0.0000$. Candidate B’s child counts total three positive and three negative, matching the parent. The common mistake avoided here is accepting child counts that do not partition the parent.

Worked Example: Comparing a Numeric Threshold With Another Split

In a hypothetical risk-screening dataset, a parent node contains eight records: four positive and four negative. A numeric predictor gives Candidate A, with children containing $(2+,0-)$ and $(2+,4-)$. A second predictor gives Candidate B, with children $(3+,1-)$ and $(1+,3-)$. Calculate each gain and select the larger one.

Parent entropy. The parent is balanced, so $H_{\mathrm{parent}}=1.000000$ bit.

Candidate A. The first child is pure, so its entropy is zero. The second child has six records, with positive proportion $2/6=1/3$ and negative proportion $4/6=2/3$. Its entropy is

H_{A,2} = -\frac13\log_2\left(\frac13\right) -\frac23\log_2\left(\frac23\right) =0.528321+0.389975 =0.918296\text{ bits}.

The child weights are $2/8=1/4$ and $6/8=3/4$. The weighted child entropy is $(1/4)(0)+(3/4)(0.918296)=0.688722$ bits. Therefore, Candidate A’s gain is $1.000000-0.688722=0.311278$ bits, or 0.3113 bits. Its children contain four positive and four negative records altogether.

Candidate B. Each child has four observations and class proportions $3/4$ and $1/4$, in opposite orders. Each child entropy is

H_{B,1}=H_{B,2} = -\frac34\log_2\left(\frac34\right) -\frac14\log_2\left(\frac14\right) =0.311278+0.500000 =0.811278\text{ bits}.

The weighted child entropy is $(4/8)(0.811278)+(4/8)(0.811278)=0.811278$ bits. Candidate B’s gain is $1.000000-0.811278=0.188722$ bits, or 0.1887 bits.

Conclusion and check. Candidate A is preferred because $0.3113>0.1887$. The counts for both candidates total four positive and four negative. In particular, Candidate A’s weighted child entropy is 0.6887 bits; it is not itself the gain. The gain is the parent entropy minus that quantity.

Worked Example: A Split With Three Categorical Children

A hypothetical parent node has six records: four positive and two negative. A categorical predictor has three levels. Candidate A sends the records to children with counts $(2+,0-)$, $(1+,1-)$, and $(1+,1-)$. Candidate B groups the levels into two children with counts $(3+,1-)$ and $(1+,1-)$. Calculate each gain, including the child weights.

Parent entropy. The class proportions are $4/6=2/3$ and $2/6=1/3$, giving

H_{\mathrm{parent}} = -\frac23\log_2\left(\frac23\right) -\frac13\log_2\left(\frac13\right) =0.918296\text{ bits}.

Candidate A. The first child is pure, so its entropy is zero and its weight is $2/6$. Each of the other two children is balanced, so each has entropy one bit and weight $2/6$. The weighted child entropy is $(2/6)(0)+(2/6)(1)+(2/6)(1)=0.666667$ bits. Candidate A’s gain is $0.918296-0.666667=0.251629$ bits, or 0.2516 bits.

Candidate B. The first child has proportions $3/4$ and $1/4$, so its entropy is $0.811278$ bits. Its weight is $4/6$. The second child is balanced, so its entropy is one bit and its weight is $2/6$. The weighted child entropy is

\frac46(0.811278)+\frac26(1) =0.540852+0.333333 =0.874185\text{ bits}.

The value $0.540852$ is only the weighted contribution of Candidate B’s first child, not the gain. Candidate B’s gain is $0.918296-0.874185=0.044111$ bits, or 0.0441 bits. Candidate A is preferred because $0.2516>0.0441$. Both candidate partitions preserve the parent totals of four positive and two negative records.

Why the Weighted Formula Works

The weighting is required because entropy describes the uncertainty for a randomly selected observation in a node. Under the empirical distribution, a randomly selected parent observation falls in child $j$ with probability $n_j/n$. Thus the average uncertainty remaining after the split is the child entropy averaged using those probabilities, not an unweighted average across the child nodes.

The corresponding reduction has a useful probabilistic interpretation. Let $C$ denote the class label and $J$ the child entered by an observation. The parent class proportions are mixtures of the child proportions: for either class $c$, $P(C=c)=\sum_j P(J=j)P(C=c\mid J=j)$. The uncertainty before observing the child is $H(C)$; after observing it, the average remaining uncertainty is $\sum_jP(J=j)H(C\mid J=j)$. Their difference is the information about the class provided by the child identity:

H(C)-\sum_jP(J=j)H(C\mid J=j) = \sum_jP(J=j)\sum_cP(C=c\mid J=j) \log_2\left(\frac{P(C=c\mid J=j)}{P(C=c)}\right).

To see the equality, expand the left side using $H(C)=-\sum_cP(C=c)\log_2P(C=c)$ and $H(C\mid J=j)=-\sum_cP(C=c\mid J=j)\log_2P(C=c\mid J=j)$. Substitute $P(C=c)=\sum_jP(J=j)P(C=c\mid J=j)$ into the first term, then collect terms by child and class. The result is the displayed sum. It is nonnegative: each inner contribution is part of a weighted Kullback–Leibler divergence between the child class distribution and the parent class distribution. Hence a valid split cannot have negative information gain in the population-count calculation, although rounding can make a tiny value appear slightly negative.

Common Mistakes and Checks

Using an unweighted average. A child with two observations must not count as much as a child with twenty. Use $n_j/n$ for each child’s weight.
Calling weighted child entropy the gain. Gain is parent entropy minus weighted child entropy. Keep these two quantities labelled separately during the calculation.
Comparing invalid child counts. For a parent with three positive and three negative records, children $(1+,2-)$ and $(1+,2-)$ total two positive and four negative records. They cannot be its children. A valid alternative is $(1+,1-)$ and $(2+,2-)$, which preserves the parent counts.
Switching logarithm bases without converting units. Base-2 logarithms give bits. Natural logarithms give entropy in nats; the numeric values differ by a factor of $\ln 2$. Do not compare a gain in bits to one in nats without conversion.
Rounding too early. Carry several decimal places through the weighted sum and subtraction, then round the final gain. Small gains can otherwise be distorted or mistaken for zero.

Key takeaway: A hand-calculated entropy gain is trustworthy only when the child counts partition the parent, each child entropy uses that child’s class proportions, and the child entropies are weighted by child size before subtraction.

Check Your Understanding

For each problem, verify the child counts against the parent before calculating or interpreting the gain. Entropies and gains are in bits; round final gains to four decimal places.

A parent has five positive and three negative records. A split gives children $(3+,1-)$ and $(2+,2-)$. Calculate the parent entropy, weighted child entropy, and gain. (Answer: parent entropy $0.9544$, weighted child entropy $0.9056$, gain $0.0488$ bits.)
A balanced parent has four positive and four negative records. Candidate X gives children $(2+,0-)$ and $(2+,4-)$; Candidate Y gives $(3+,1-)$ and $(1+,3-)$. Which candidate has greater gain, and what is that gain? (Answer: Candidate X, $0.3113$ bits; Candidate Y has $0.1887$ bits.)
A parent has four positive and two negative records. A three-child split gives counts $(2+,0-)$, $(1+,1-)$, and $(1+,1-)$. Calculate the parent entropy, weighted child entropy, and gain. (Answer: parent entropy $0.9183$, weighted child entropy $0.6667$, gain $0.2516$ bits.)
A balanced parent has six positive and six negative records. Candidate A produces children $(4+,2-)$ and $(2+,4-)$. Candidate B produces $(3+,3-)$ and $(3+,3-)$. Calculate the gain for each and choose the preferred split. (Answer: Candidate A has gain $0.0817$ bits; Candidate B has gain $0.0000$ bits. Prefer Candidate A.)
A parent has four positive and two negative records. Candidate P has children $(3+,1-)$ and $(1+,1-)$. Calculate its weighted child entropy and gain, and explain why $0.5409$ is not the gain. (Answer: weighted child entropy $0.8742$ bits; gain $0.0441$ bits. $0.5409$ is only the first child’s weighted entropy contribution.)

What You'll Learn