What Impurity Importance Measures
A random forest can report which predictors contributed most to its tree splits. One common measure, impurity importance, adds up the impurity reductions credited to each predictor across the forest. It describes how much the fitted trees used a predictor to make locally purer child nodes. It does not, by itself, establish that the predictor has a strong relationship with the outcome in new data.
The measure is useful as a summary of how a forest was constructed, but its ranking can be misleading. In particular, a predictor with many possible split points has more opportunities to achieve a large improvement by chance than a predictor with only a few possible splits. This can favor continuous predictors and categorical predictors with many levels, even when their relationship with the outcome is weak or absent.
Calculating a Split’s Contribution
For a regression tree, a common impurity measure is the mean squared deviation of the response from its node mean. For a classification tree, a common measure is Gini impurity, which is the probability that two labels drawn independently from the node’s observed class proportions differ. The formulas below assume that the split creates two nonempty child nodes and that all observations receive equal weight. Weighted observations require corresponding weighted counts and proportions.
Here, \(t\) denotes a node, \(n_t\) is its number of observations, \(\bar{y}_t\) is the mean response in that node, and \(p_{tk}\) is the proportion of class \(k\) in the node. If the children are \(L\) and \(R\), with sizes \(n_L\) and \(n_R\), the decrease is:
A forest commonly credits this decrease to the variable used at the split, with node-size weighting or an equivalent scaling. For example, a tree-level contribution can be written as \((n_t/N)\Delta_t\), where \(N\) is the number of observations used for that tree. Summing such contributions over a tree, and then aggregating over trees, gives an impurity-based importance measure. Software can use a different scaling, such as a raw sum or an average over trees. Therefore, compare scores only when the forest settings and importance definition are the same.
Worked Example: A Regression Split’s Improvement
A hypothetical tree has eight observations with responses \(1,1,2,2,7,7,8,8\). A continuous predictor separates the first four observations from the last four. Calculate the parent impurity, the child impurities, and the split’s reduction. Use mean squared deviation as the regression impurity.
Method. Calculate each node’s mean and mean squared deviation, then subtract the weighted child impurity from the parent impurity. This method avoids comparing unweighted child impurities when the child sizes differ.
The parent mean is \((1+1+2+2+7+7+8+8)/8=36/8=4.5\). The sum of squared deviations is \(2(1-4.5)^2+2(2-4.5)^2+2(7-4.5)^2+2(8-4.5)^2=2(12.25)+2(6.25)+2(6.25)+2(12.25)=74\). Thus the parent impurity is \(74/8=9.25\).
The left child mean is \((1+1+2+2)/4=1.5\). Its sum of squared deviations is \(2(1-1.5)^2+2(2-1.5)^2=1\), so its impurity is \(1/4=0.25\). The right child mean is \((7+7+8+8)/4=7.5\); its sum of squared deviations is also \(1\), so its impurity is \(0.25\).
The weighted child impurity is \((4/8)(0.25)+(4/8)(0.25)=0.25\). Therefore, the reduction is \(9.25-0.25=9.00\). As a check, the total within-child squared deviation is \(1+1=2\), and \((74-2)/8=9.00\), matching the impurity calculation. The split receives a large contribution because it separates the lower and higher responses.
For this regression impurity, the size-weighted reduction has a useful equivalent form: it equals the decrease in total within-node sum of squares divided by the full tree sample size. The equality follows by multiplying each node’s mean squared deviation by its node size:
The identity holds because \(n_t I_t=\operatorname{SSE}_t\), and likewise for each child. It explains why a split’s contribution reflects both its reduction in within-node variation and the fraction of the tree’s observations affected. It does not imply that impurity importance is an unbiased estimate of a predictor’s general predictive value.
Worked Example: A Classification Split’s Gini Reduction
A hypothetical classification node contains ten cases: five positive and five negative. A candidate split produces a left child with three positive and one negative case, and a right child with two positive and four negative cases. Calculate the parent and child Gini impurities and the reduction.
Method. Calculate \(1-\sum_k p_k^2\) in each node, then average child impurities using their sample-size proportions. This avoids treating a small child as if it had the same weight as a large child.
The parent class proportions are \(5/10\) and \(5/10\), so its Gini impurity is \(1-(5/10)^2-(5/10)^2=1-0.25-0.25=0.50\).
The left child proportions are \(3/4\) and \(1/4\). Its Gini impurity is \(1-(3/4)^2-(1/4)^2=1-9/16-1/16=6/16=0.375\). The right child proportions are \(2/6=1/3\) and \(4/6=2/3\). Its Gini impurity is \(1-(1/3)^2-(2/3)^2=1-1/9-4/9=4/9\).
The weighted child impurity is \((4/10)(0.375)+(6/10)(4/9)=0.15+4/15=25/60=5/12\). The decrease is \(1/2-5/12=1/12\), approximately \(0.0833\). The decrease is positive, as expected: the children are, on average, more class-pure than the parent. The exact check is \(0.50-0.4167\approx0.0833\).
Why More Split Opportunities Can Inflate Importance
At a node, a tree-growing algorithm typically evaluates eligible candidate splits and selects one that gives a large impurity reduction. A numerical predictor with many distinct values can offer many threshold locations. A categorical predictor with many levels can also offer many ways to divide its levels, depending on the tree implementation. By contrast, a binary predictor usually offers only one distinct division at a node.
Even if every candidate split is unrelated to the outcome in the population, the observed sample can contain accidental patterns. Searching more candidates makes it more likely that at least one candidate will line up with such a pattern. The tree then selects that candidate, and its observed reduction is credited to the selected predictor.
This reasoning describes a selection advantage, not a guarantee that every high-cardinality variable will rank highly. The actual ranking also depends on the observed data, which predictors are eligible at each node, tree depth and stopping rules, and the forest implementation. Impurity importance can accumulate contributions from many splits, including splits deeper in trees.
Worked Example: A Chance Gini Gain From Many Candidate Splits
Suppose eight hypothetical cases have four positive and four negative outcomes. A predictor consisting of eight unique identifiers has no population relationship with the outcome, but the tree can test one-versus-rest splits. In the observed sample, one identifier belongs to a positive case. Calculate the Gini reduction from isolating that case.
Method. Compare the parent Gini impurity with the size-weighted child impurity for the proposed split. This illustrates how a split can improve the training-node impurity even when the predictor is only an identifier.
The parent has class proportions \(4/8\) and \(4/8\), so its Gini impurity is \(1-(4/8)^2-(4/8)^2=0.50\). The isolated case is positive, so its child has impurity \(1-1^2-0^2=0\).
The remaining seven cases contain three positive and four negative outcomes. Their Gini impurity is \(1-(3/7)^2-(4/7)^2=1-9/49-16/49=24/49\). The weighted child impurity is \((1/8)(0)+(7/8)(24/49)=3/7\). Therefore the reduction is \(1/2-3/7=1/14\), approximately \(0.0714\). Checking, \(0.5000-0.4286=0.0714\).
The reduction is a real property of this observed split, but it is not evidence that the identifier will predict outcomes in new cases. A predictor with many possible divisions has more opportunities to find a favorable split like this one. An identifier should therefore prompt scrutiny of the data design and candidate split structure rather than an immediate substantive interpretation.
Checks Before Interpreting a Ranking
An impurity ranking is most informative when treated as a description of the fitted forest, not as a definitive ordering of the predictors’ underlying importance. Before reporting or acting on the ranking, examine how each predictor could be split and how the importance was computed.
Check the number of distinct numerical values and factor levels. A predictor with many candidate thresholds or level groupings may have a selection advantage.
Record whether the software reports impurity importance, how it aggregates contributions across trees, and whether scores are scaled. Do not compare values from different definitions as if they shared a common scale.
Check the split rule, tree size controls, sampling approach, and which variables were eligible at each split. These choices affect which opportunities the trees had to accumulate impurity reductions.
Impurity importance records split use in the fitted forest. On its own, it does not establish causation, a stable ordering, or improved predictive performance on unseen data.
A common exam trap is to treat the largest impurity-importance value as proof that the predictor is intrinsically the most useful. For example, if a continuous predictor offers many candidate thresholds while a binary predictor offers one, comparing only their accumulated impurity reductions ignores the unequal search opportunities. The correction is to describe the ranking as a property of this fitted forest and check the candidate split counts and importance convention before drawing a broader conclusion.
Check Your Understanding
Work each problem using the stated impurity measure and equal observation weights.
- A regression node has six responses \(2,2,2,8,8,8\). A split separates the three lower from the three higher responses. What are the parent impurity, weighted child impurity, and reduction using mean squared deviation? (Answer: parent impurity \(9\), weighted child impurity \(0\), reduction \(9\).)
- A classification node has six cases, three in each of two classes. A split creates two children, each containing two cases from one class and one from the other. Calculate the Gini reduction. (Answer: parent \(0.5000\), weighted child impurity \(4/9\), reduction \(1/18\approx0.0556\).)
- A binary predictor has one possible split at a node, while a continuous predictor has 15 distinct values and can be split at multiple thresholds. If both have no population relationship with the outcome, which has more opportunity to receive a large chance gain, and why? (Answer: the continuous predictor, because the tree selects among more candidate thresholds and can choose the largest observed reduction.)
- A forest’s impurity importance gives a high score to a factor with 30 levels. State one reason the score may be inflated and one check to perform before interpreting it. (Answer: many level groupings can provide more candidate splits and chance gains; inspect the factor’s split opportunities and confirm the importance definition and forest settings.)
- A predictor has the largest impurity importance in a fitted forest. Does this alone establish that it causes the outcome or improves prediction on new data? (Answer: no; it summarizes accumulated split improvement in the fitted forest and alone establishes neither causation nor out-of-sample predictive improvement.)