Mean Decrease In Impurity Importance

What Mean Decrease in Impurity Measures

A random forest can rank predictors by how much they improve the splits used to build its trees. At each split, the tree divides a node into child nodes chosen to have lower impurity than the parent. Mean decrease in impurity (MDI) accumulates those reductions for each predictor, then averages the accumulated contributions across trees. It is a measure of how the fitted forest used predictors while constructing its trees.

The impurity measure depends on the task and the tree-fitting criterion. For regression, a common choice is within-node squared error, expressed as mean squared error or as a weighted sum of squared deviations. For classification, a common choice is Gini impurity. The calculations below use these criteria; a tree fitted with another criterion, such as entropy for classification, uses that criterion’s impurity reductions instead. No parametric distribution assumption is required for these sample-based calculations.

Definition: Let $t$ be a node with total case weight $W_t$, impurity $I(t)$, and children $c$. The weighted impurity decrease at that split is $$ \Delta(t)=\frac{W_t}{W}\left[I(t)-\sum_c\frac{W_c}{W_t}I(c)\right], $$ where $W$ is the total case weight used to scale the tree. For a tree, the raw importance of predictor $j$ is the sum of $\Delta(t)$ over its split nodes that use $j$. The forest importance is typically the average of these tree-level sums over the $B$ trees. A commonly reported relative importance divides each predictor’s forest score by the sum of all predictor scores, so the relative scores sum to 1 (or 100%).

With unweighted observations, $W_t$ is the number of observations at node $t$. If a tree uses bootstrap samples, its node counts and scaling are based on the observations or weights used to fit that tree. Software can differ in its precise handling of weights and normalization, so an importance calculation should be interpreted in light of the implementation’s convention. The core calculation is the weighted improvement in the tree’s chosen impurity criterion.

Impurity Measures and a Useful Identity

For regression, let the response values at node $t$ have mean $\bar y_t$. Under mean squared error impurity, $I(t)=\frac{1}{W_t}\sum_{i\in t}w_i(y_i-\bar y_t)^2$, where $w_i$ is the case weight. The decrease at the split is therefore the parent mean squared error minus the weighted average of the children’s mean squared errors, multiplied by $W_t/W$. For classification with $K$ classes and class proportions $p_{tk}$ at node $t$, Gini impurity is $I(t)=1-\sum_{k=1}^K p_{tk}^2$. A node is pure when all its cases belong to one class, giving Gini impurity zero.

The weight on a node’s impurity change matters. A large improvement in a node containing only a few cases should not count as much as the same-sized improvement in a node containing most of the tree’s cases. Multiplying by $W_t/W$ accounts for that difference.

Why the node formula works: Multiplying the bracketed difference by $W_t/W$ gives $$ \Delta(t)=\frac{W_t I(t)-\sum_c W_c I(c)}{W}. $$ Thus the contribution is the parent’s total weighted impurity minus the children’s total weighted impurity, scaled by the tree’s total weight. If these contributions are added across the whole tree, an intermediate node’s impurity appears once as a child and once as a parent, with opposite signs. Those terms cancel. The remaining difference is the root’s weighted impurity minus the weighted impurity at the terminal leaves, divided by $W$. This explains why the accumulated split reductions track the tree’s overall impurity reduction.

The identity also clarifies what is being ranked: the predictors receive credit for impurity reduction at the particular nodes where the fitted tree selected them. A predictor used near the root can receive a large contribution because that node contains many cases, but a predictor used in several lower nodes can also accumulate substantial importance.

Worked Calculations

Worked Example: Regression Importance in One Tree

Hypothetical scenario. A regression tree is fitted to eight claim-processing times, measured in hours, with observed responses $1,1,3,3,5,5,7,7$. The tree first splits on predictor X into the first four and last four responses. It then splits each group on predictor Z into two equal-response pairs. Use mean squared error impurity, equal observation weights, and $W=8$. No probability distribution is assumed; the listed responses are the complete training data for this example. Find each predictor’s raw contribution and relative importance.

Method. Calculate the impurity decrease at each split, weighting each node’s change by its fraction of all eight observations. The common mistake is to add unweighted node impurity decreases, which would give small descendant nodes too much influence.

Root and X split. The root mean is $(1+1+3+3+5+5+7+7)/8=32/8=4$. Its sum of squared deviations is $4(4-1)^2+4(4-5)^2$, or, directly, $2(1-4)^2+2(3-4)^2+2(5-4)^2+2(7-4)^2=40$. The root mean squared error is $40/8=5$. Each X child has four responses with mean 2 or 6 and sum of squared deviations 4, so each child has mean squared error $4/4=1$. The weighted child impurity is $1$, and the root decrease is $5-1=4$. Since the root contains all eight observations, X receives a weighted contribution of $(8/8)(4)=4$.

Z splits. In either X child, the parent mean squared error is 1. Its two Z children are constant-response pairs, so both have mean squared error 0. Each X child contains four of the eight observations, so each Z split contributes $(4/8)(1-0)=0.5$. Across the two Z splits, Z receives $0.5+0.5=1$.

The raw tree scores are therefore X $=4$ and Z $=1$, with total $5$. Normalizing gives X $=4/5=0.8$ and Z $=1/5=0.2$. As a check, the total weighted decrease is 5, equal to the root mean squared error of 5 because all terminal nodes have zero impurity. The relative importances sum to 1.

Worked Example: Gini Importance in a Classification Tree

Hypothetical scenario. A classification tree is fitted to ten policy-service records, each labelled either “routine” or “review.” At the root there are five records of each class. The root splits on predictor C into two groups of five: the first has four routine and one review, and the second has one routine and four review. Each of those groups is then split on predictor D into pure child nodes. Use Gini impurity and equal case weights. Calculate the raw and relative contributions. The class labels are fixed observations; no class-probability model is assumed.

Root and C split. The root Gini impurity is $1-(5/10)^2-(5/10)^2=0.5$. Each C child has class proportions $4/5$ and $1/5$, in one order or the other, so its Gini impurity is $1-(4/5)^2-(1/5)^2=1-16/25-1/25=8/25=0.32$. The weighted average of the two child impurities is $(5/10)(0.32)+(5/10)(0.32)=0.32$. C’s root contribution is therefore $0.5-0.32=0.18$.

D splits. Each C child has impurity 0.32 and is split into pure nodes with impurity 0. Each D split contributes $(5/10)(0.32-0)=0.16$, because each parent contains five of the ten observations. D’s total contribution is $0.16+0.16=0.32$.

The total raw importance is $0.18+0.32=0.50$. The relative importance of C is $0.18/0.50=0.36$, and that of D is $0.32/0.50=0.64$. The check is that the total decrease, 0.50, equals the root Gini impurity because every terminal node is pure. D ranks higher even though C made the root split: MDI sums reductions over all split nodes, not just the root.

From Tree Scores to Forest Scores

Each tree gives a predictor credit for the weighted impurity decreases at its own split nodes. The forest score combines these tree-level credits. The following example illustrates a common convention: calculate the raw sum for each predictor within each tree, average each predictor’s sum over the trees, and then normalize the resulting averages. All node contributions below are already weighted decreases on the scale used for their respective trees.

Worked Example: Aggregating Importance Across Three Trees

Hypothetical scenario. A three-tree regression forest has predictors A, B, and C. In tree 1, the weighted decreases at nodes using A are 0.25 and 0.15, and the decrease at a node using B is 0.10. In tree 2, A contributes 0.10 and C contributes 0.30. In tree 3, B contributes 0.20 and C contributes 0.20. Calculate the forest raw scores and relative importances. These are supplied tree-level impurity reductions; no response distribution is assumed.

Method. First sum within each tree and predictor, treating a predictor not used in a tree as contributing zero. Then average over all three trees. Do not average only over the trees in which a predictor appears, because that would give predictors used rarely an inflated score.

Tree 1 gives A $0.25+0.15=0.40$, B $0.10$, and C $0$. Tree 2 gives A $0.10$, B $0$, and C $0.30$. Tree 3 gives A $0$, B $0.20$, and C $0.20$. Averaging over three trees gives A $(0.40+0.10+0)/3=0.50/3$, B $(0.10+0+0.20)/3=0.30/3$, and C $(0+0.30+0.20)/3=0.50/3$. The total average is $1.30/3$.

Normalizing cancels the common divisor of 3. A’s relative importance is $0.50/1.30=0.3846$, B’s is $0.30/1.30=0.2308$, and C’s is $0.50/1.30=0.3846$, to four decimal places. The scores sum to $1.0000$, as required. A and C tie under this measure, while B has the smaller accumulated split contribution.

Interpreting the Ranking Carefully

MDI is a property of a fitted forest and its split criterion. A large score means that, within these fitted trees, splits using that predictor accumulated substantial weighted impurity decreases. It does not by itself show that the predictor has a causal effect, that its association has a particular direction, or that its contribution is statistically significant. The score is not a regression coefficient, a hypothesis test, or an estimate of the change in response when a predictor changes.

It is also distinct from importance defined by a change in prediction error. Error-based approaches assess how prediction performance changes when a predictor is disrupted, often by permuting its values and comparing the resulting error with a baseline. MDI instead reads the reductions recorded during tree construction. Thus the two quantities answer different questions: one summarizes split-based impurity reduction; the other assesses a change in predictive error. An MDI ranking should not be described as an out-of-bag error increase.

Several practical details can affect the numerical ranking. The chosen impurity criterion matters, as do the available candidate predictors at each split, the tree-growing and stopping rules, and the forest’s sampling and weighting conventions. A predictor never selected for a split has raw MDI of zero under the fitted forest, but that alone does not establish that it has no relationship with the response. Relative scores are also compositional: increasing one predictor’s share changes other predictors’ percentages even if their raw scores stay fixed.

For actuarial modeling, this distinction is useful when summarizing a forest used for a hypothetical claim-frequency or service-time problem. MDI gives a compact account of which predictors supported the fitted partitions under the specified tree criterion. It does not replace checking predictive performance with suitable validation, nor does a high ranking establish that a predictor should be used for a particular decision. Interpretation should remain tied to the fitted model and the data used to build it.

Key takeaway: Calculate each split’s weighted impurity reduction, sum reductions by predictor within trees, and aggregate across trees. Normalize only after aggregation if relative scores are wanted. MDI ranks accumulated split improvements; it is not a measure of changes in prediction error.

These definitions and calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. On an exam-style calculation, identify the impurity criterion and node weights first, keep raw and normalized scores distinct, and verify that the normalized importances sum to 1 (or 100%).

Check Your Understanding

For each problem, use the stated impurity criterion and weighting convention. Answers are given at the end of each question for checking.

A classification node contains 12 observations, six in each class. A split creates two children of six observations each, with class counts 5 and 1 in one child and 1 and 5 in the other. Using Gini impurity, calculate the weighted impurity decrease at the split. (Answer: 0.2222)
A regression node contains 10 equally weighted observations. Its parent sum of squared deviations is 50; its two children have sums of squared deviations 18 and 12. Using mean squared error impurity and scaling by the total of 10 observations, find the weighted decrease. (Answer: 2)
Across two trees, predictor A has raw tree scores 0.24 and 0.06, B has 0.16 and 0, and C has 0 and 0.18. Average each score over both trees, then normalize the averages to sum to 1. (Answer: A = 0.4688, B = 0.2500, C = 0.2813)
A predictor has a large normalized MDI score in a fitted forest. What does that establish about the fitted trees, and what does it not establish about prediction error or causality? (Answer: It establishes that the predictor received a large share of the forest’s accumulated weighted split-impurity reductions; it does not establish a change in prediction error, a causal effect, or statistical significance.)
In a two-tree forest, tree 1 has weighted decreases of 0.30 for M and 0.08 and 0.12 for N. Tree 2 has decreases of 0.25 for Q and 0.05 for M. Treat unused predictors as zero, average each raw score over both trees, and normalize. (Answer: M = 0.4375, N = 0.2500, Q = 0.3125)

What You'll Learn