Why Use Two Measures of Importance?
A random forest can report how important each predictor is, but “importance” depends on what is measured. OOB permutation importance asks how much prediction performance worsens when a predictor’s OOB values are shuffled. Mean decrease in impurity (MDI) summarizes how much that predictor’s splits reduce impurity while the trees are being fitted. One measures a change in predictive loss; the other summarizes split improvements.
The measures need not give the same ranking, even for the same fitted forest. They use different information and answer different questions. A comparison is meaningful only after stating the loss or impurity criterion, how tree-level quantities are averaged, and whether reported scores have been rescaled.
OOB Permutation Importance
For a regression forest, one possible loss is mean squared error (MSE). For each tree, calculate the MSE on its OOB rows before and after permuting one predictor. Subtract the original MSE from the permuted MSE, then average those differences over trees. A large positive value means that shuffling the predictor substantially worsened OOB predictions under this procedure.
For classification, a loss might be misclassification rate or a probability-based loss. The choice matters: the resulting score describes the deterioration in that particular measure of predictive performance. In the examples below, tree-level differences are averaged with equal weight across trees. Other implementations may pool OOB losses or use ensemble-level OOB predictions; report the convention when comparing scores.
The permutation is performed among the OOB rows for a tree. It breaks the observed pairing between that predictor’s values and the outcomes in those rows, while leaving the tree’s fitted structure and other predictor values unchanged. This is a predictive diagnostic, not a causal effect.
Worked Example: Permutation Importance for a Regression Forest
A hypothetical two-tree regression forest is evaluated using squared error. Each tree has four OOB observations. The table gives the sum of squared errors (SSE) before permutation and after separately permuting predictors \(A\) and \(B\). Calculate the average OOB permutation importance for each predictor and rank them.
| Tree | Original OOB SSE | SSE after permuting A | SSE after permuting B |
|---|---|---|---|
| 1 | 4 | 12 | 8 |
| 2 | 2 | 18 | 2 |
Method. Convert each SSE to MSE by dividing by the four OOB observations, find the increase for each permutation, and average the two tree-specific increases.
For tree 1, the original MSE is \(4/4=1\). After permuting \(A\), it is \(12/4=3\), an increase of \(3-1=2\). After permuting \(B\), it is \(8/4=2\), an increase of \(2-1=1\).
For tree 2, the original MSE is \(2/4=0.5\). After permuting \(A\), it is \(18/4=4.5\), an increase of \(4.5-0.5=4\). After permuting \(B\), it remains \(2/4=0.5\), an increase of \(0.5-0.5=0\).
Thus \(A\) ranks above \(B\) by OOB permutation importance. The scores are increases in MSE, so they are measured in squared response units. Common mistake: comparing an SSE increase from one tree directly with an MSE increase from another. The calculation first uses the stated per-tree MSE convention, so each difference is on the same scale.
Mean Decrease in Impurity
A tree chooses splits to reduce node impurity. For regression, a common impurity is the mean squared deviation from the node mean (equivalently, a variance using the divisor equal to the node’s case count). For classification, common choices include Gini impurity and entropy. The impurity criterion is determined by the tree-fitting procedure; it is not necessarily the same as the loss used for OOB permutation importance.
Suppose a split at node \(t\) sends \(n_L\) of its \(n_t\) fitting cases to the left child and \(n_R\) to the right child. Let \(I(t)\), \(I(t_L)\), and \(I(t_R)\) be the node impurities. If the tree has \(n_b\) fitting cases, the split’s weighted decrease is
The tree’s contribution for predictor \(j\) is the sum of \(\Delta_b(t)\) over its splits using \(j\). MDI averages that contribution over the forest’s trees. Here, as in many implementations, the fitting sample for a tree is its bootstrap sample, with the implementation’s chosen convention for counting repeated bootstrap rows. Software may subsequently normalize or rescale the scores, so raw values from different implementations are not automatically comparable.
For squared-error regression, the weighted impurity reduction also has a useful interpretation. If \(I(t)=\mathrm{SSE}(t)/n_t\), then
The equality follows because the weighted child impurity is \(\frac{n_L}{n_t}\frac{\mathrm{SSE}(t_L)}{n_L} +\frac{n_R}{n_t}\frac{\mathrm{SSE}(t_R)}{n_R}\), which simplifies to the sum of the child SSEs divided by \(n_t\). MDI therefore accumulates training-sample error reductions, weighted by the tree’s fitting-sample size.
Worked Example: Calculate MDI and Compare the Rankings
For the same hypothetical forest as in the preceding example, suppose the weighted squared-error impurity contributions, already calculated using each tree’s fitting sample, are as follows. For tree 1, splits using \(A\) contribute \(8/4=2\), and splits using \(B\) contribute \(2/4=0.5\). For tree 2, the corresponding contributions are \(4/4=1\) for \(A\) and \(12/4=3\) for \(B\). Calculate the MDI scores and compare their ranking with the OOB permutation ranking.
Method. Average each predictor’s weighted contribution across the two trees:
By MDI, \(B\) ranks above \(A\), whereas the OOB permutation scores were \(3\) for \(A\) and \(0.5\) for \(B\). The rankings differ for this fitted forest. MDI records weighted reductions made by splits on the fitting samples; the permutation calculation records deterioration in OOB MSE after shuffling. Their different rankings are not an arithmetic contradiction.
The scales also differ: these MDI values are impurity reductions under the stated squared-error convention, while the permutation values are increases in OOB MSE. Do not subtract one kind of score from the other or interpret their numerical sizes as directly comparable. Common mistake: assuming that the predictor used in splits with the largest total impurity reduction must also cause the largest OOB performance loss. The measures assess different quantities.
Classification Example: The Scoring Rule Matters
For a classification forest, OOB permutation importance depends on the chosen classification loss. If the loss is misclassification rate, only whether each prediction is correct matters. A change in a predicted class probability that does not change the predicted class makes no difference to this loss. MDI, in contrast, is based on the tree’s impurity criterion, such as Gini impurity, and the reductions at its splits.
Worked Example: Compare Classification Rankings
A hypothetical two-tree classifier has 10 OOB observations per tree. Using misclassification rate, tree 1 makes 2 errors originally, 6 after permuting \(C\), and 3 after permuting \(D\). Tree 2 makes 1 error originally, 5 after permuting \(C\), and 2 after permuting \(D\). Separately, the weighted Gini-decrease contributions to MDI are \(0.08\) and \(0.04\) for \(C\), and \(0.12\) and \(0.10\) for \(D\), in trees 1 and 2 respectively. Calculate both rankings.
OOB permutation importance. For tree 1, the original error rate is \(2/10=0.2\). Permuting \(C\) gives \(6/10=0.6\), an increase of \(0.6-0.2=0.4\); permuting \(D\) gives \(3/10=0.3\), an increase of \(0.3-0.2=0.1\).
For tree 2, the original rate is \(1/10=0.1\). Permuting \(C\) gives \(5/10=0.5\), an increase of \(0.4\); permuting \(D\) gives \(2/10=0.2\), an increase of \(0.1\). Averaging across trees gives \((0.4+0.4)/2=0.4\) for \(C\) and \((0.1+0.1)/2=0.1\) for \(D\). Therefore, \(C\) ranks higher by OOB permutation importance.
MDI. Average the supplied weighted Gini-decrease contributions:
Therefore, \(D\) ranks higher by MDI. The two rankings disagree: the OOB misclassification-rate calculation favors \(C\), while the Gini-based split summary favors \(D\). Common mistake: treating misclassification rate and Gini impurity as interchangeable. They are distinct criteria, so their importance scores need not agree.
Interpreting Differences and Small Scores
A disagreement between the measures is a reason to examine how each score was constructed, not a reason to declare one calculation wrong. Check the following before interpreting a ranking:
- Identify the target quantity. Permutation importance measures change in a specified OOB loss; MDI summarizes weighted reductions in the trees’ fitting impurity.
- Confirm the aggregation. Tree averages, pooled OOB losses, and ensemble-level OOB predictions are different conventions. Comparisons require like-for-like calculations.
- Check the scale. Raw scores and normalized scores can have different units and magnitudes. A normalized score may preserve a ranking while changing every reported value.
- Interpret negative permutation estimates carefully. A negative estimate means that the selected loss happened to be smaller after permutation under the calculation used. It does not establish that the predictor harms prediction; finite OOB samples and variation in the permutations can produce small negative differences.
Neither measure, by itself, establishes that changing a predictor would change the outcome. Permutation importance describes predictive deterioration under a particular shuffle and loss; MDI describes how a fitted forest used predictors to reduce impurity. Their interpretations should remain tied to those definitions.
Worked Example: A Small Negative Permutation Estimate
For a predictor \(E\), a three-tree regression forest has tree-level OOB MSE increases of \(0.12\), \(-0.03\), and \(0.00\) after permutation. Using the equal-tree average, calculate the importance and interpret its sign.
Calculation. Add the three changes and divide by three:
The average estimate is positive, \(0.03\) MSE units, even though one tree’s change is negative. If instead the three changes had been \(0.02\), \(-0.05\), and \(0.00\), their average would be \(-0.03/3=-0.01\). That negative estimate would mean that the observed average OOB MSE was slightly lower after permutation under this calculation. It would not prove that \(E\) is harmful or that its true predictive contribution is negative. Common mistake: interpreting one tree’s negative change as the forest’s overall importance; average the stated tree-level changes first.
Check Your Understanding
For each problem, use the stated averaging and scoring conventions.
- A two-tree regression forest has OOB MSE increases of \(0.6\) and \(0.2\) for predictor \(A\), and \(0.1\) and \(0.5\) for predictor \(B\). Calculate both permutation scores and state the ranking. (Answer: \(A=0.4\), \(B=0.3\); \(A\) ranks higher.)
- A classification tree has 20 OOB cases and 3 original misclassifications. After permuting \(Q\), it has 7 misclassifications. Calculate the tree’s OOB permutation importance using misclassification rate. (Answer: \(7/20-3/20=0.2000\).)
- A regression tree’s fitting sample has \(n_b=8\). At a split using \(R\), the parent SSE is 40 and the two child SSEs are 24 and 8. Calculate this split’s weighted impurity reduction under the squared-error convention. (Answer: \((40-24-8)/8=1\).)
- For predictor \(S\), two trees contribute MDI values \(0.10\) and \(0.04\); for \(T\), they contribute \(0.06\) and \(0.14\). What are the average MDI scores, and which predictor ranks higher? (Answer: \(S=0.07\), \(T=0.10\); \(T\) ranks higher.)
- A predictor has OOB MSE increases of \(0.04\), \(-0.10\), and \(0.00\) over three trees. Calculate the equal-tree average and state what the sign does and does not show. (Answer: \((0.04-0.10+0.00)/3=-0.0200\); the estimated average loss decreased slightly after permutation, but this alone does not establish that the predictor is harmful.)