Why Correlated Predictors Complicate Importance
A tree ensemble can predict well while giving an unstable or misleading ranking of its input variables. This often occurs when two predictors carry much of the same information. A tree may split on one predictor rather than the other, and a prediction may remain accurate even if one of them is disrupted because the other still provides a useful signal. The model's predictive success and the apparent importance of each individual variable are therefore different questions.
This distinction matters in risk modeling. For example, two hypothetical measures of property condition might be strongly associated and both help predict claim severity. An importance ranking may describe how a fitted ensemble used those measures, but it does not establish that one measure is independently informative, causal, or more valuable in every population.
Split-Based Importance
At a node \(t\), let \(N_t\) be the number of training observations reaching the node, and let \(N_L\) and \(N_R\) be the counts reaching its left and right child nodes. Let \(I(t)\) denote the node impurity. For a regression tree, a common choice is the within-node mean squared error; for a classification tree, common choices include the Gini impurity and entropy. The weighted impurity decrease from splitting node \(t\) is
where \(N\) is the number of training observations used for the tree. For predictor \(j\), a common forest importance is the sum of \(\Delta I_t\) over the nodes that split on \(j\), averaged across trees. Some implementations use different weighting or normalization conventions, so an interpretation should identify the reported measure rather than assume every software package calculates it identically.
The weighting in the formula matters. A decrease at a node reached by many observations contributes more than the same decrease at a small node. The impurity at the parent is reduced by the weighted child impurities; the decrease is not simply the difference between the parent impurity and one child impurity.
Worked Example: Normalizing Split-Based Importance
A hypothetical regression forest for claim-cost prediction reports the following summed, observation-weighted decreases in training mean squared error. Predictors \(A\) and \(B\) have sample correlation 0.96; \(C\) is weakly correlated with them. The ensemble's supplied node summaries are \(A:18+12+6\), \(B:16+8+6\), and \(C:5+3+2\). These are deterministic hypothetical summaries, not observations from an assumed probability distribution. No interest rate or mortality assumption applies.
Method. Add the weighted decreases for each predictor, then divide each total by the sum across predictors to obtain normalized shares.
For \(A\), the total is \(18+12+6=36\). For \(B\), it is \(16+8+6=30\). For \(C\), it is \(5+3+2=10\). The overall total is \(36+30+10=76\).
The normalized shares are \(36/76=0.473684\), \(30/76=0.394737\), and \(10/76=0.131579\). As percentages to one decimal place, they are \(47.4\%\), \(39.5\%\), and \(13.2\%\). Their displayed values total \(100.1\%\) because of rounding; before rounding, the shares sum to 1.
Conclusion and trap avoided. \(A\) ranks ahead of \(B\) by this split-based measure. The result does not demonstrate that \(A\) contains much more predictive information than \(B\): with correlation 0.96, the trees can allocate shared signal between them. A common mistake is to treat the normalized shares as percentages of unique or causal contribution. They are shares of the chosen split-based score, not a decomposition of unique predictive information.
Permutation Importance
Permutation importance evaluates the effect of disrupting a predictor on a specified loss. First calculate the model's baseline loss on an evaluation data set that was not used to fit the model. Then shuffle the values of predictor \(j\) across those rows, keep the other predictors unchanged, and calculate the loss again. The difference is the permutation importance for that shuffle:
Here \(L\) might be mean squared error for regression or a suitable classification loss. A positive value means that this permutation worsened performance. Because shuffles vary, a more stable report averages the difference over multiple independent shuffles. The evaluation observations and loss should be held constant when comparing predictors.
Worked Example: Comparing Individual and Grouped Permutation
A hypothetical regression forest is evaluated on the same held-out observations for each calculation. Its baseline mean squared error is 4.0. Predictors \(A\) and \(B\) are highly correlated. Across three shuffles, the recorded losses are 4.3, 4.5, and 4.4 when \(A\) is shuffled; 4.2, 4.4, and 4.3 when \(B\) is shuffled; and 6.1, 5.9, and 6.0 when \(A\) and \(B\) are shuffled together using the same row permutation for both. These are hypothetical diagnostics; no probability distribution, interest rate, or mortality assumption is imposed.
Method. For each case, average the shuffled losses and subtract the baseline loss of 4.0.
For \(A\), the average shuffled loss is \((4.3+4.5+4.4)/3=13.2/3=4.4\), so its mean permutation importance is \(4.4-4.0=0.4\). For \(B\), the average is \((4.2+4.4+4.3)/3=12.9/3=4.3\), giving \(4.3-4.0=0.3\). For the joint permutation, the average is \((6.1+5.9+6.0)/3=18.0/3=6.0\), giving \(6.0-4.0=2.0\).
Conclusion and trap avoided. Shuffling either predictor alone has a small effect, while disrupting the pair together has a much larger effect. This is consistent with the pair carrying shared predictive information: after one predictor is shuffled, the other can still help. The joint calculation is a useful complement, not the sum of the individual importances; here \(0.4+0.3=0.7\), which is not the joint increase of \(2.0\). Adding individual scores as if they were non-overlapping contributions is the mistake to avoid.
Why Correlation Changes the Interpretation
Permutation importance has a specific limitation when predictors are correlated. A marginal permutation shuffles one column independently of the others. It breaks the relationship between that column and the outcome, but also breaks its observed relationship with the other predictors. The shuffled rows may therefore contain combinations of predictor values that were rare or absent in the evaluation data. The resulting loss change describes this intervention on the input table, not necessarily the effect of removing a predictor while preserving the data's dependence structure.
Redundant predictors can also mask one another's individual importance. If the model can rely on either of two correlated variables, shuffling one may leave a useful substitute intact. Conversely, a permutation can produce implausible combinations that cause a large loss increase. Thus a low or high individual permutation score should be interpreted in light of the correlation structure and the fitted model.
A grouped permutation shuffles a set of related predictors together, applying the same row permutation to each member of the group. This preserves the within-group combinations from the original rows while disrupting their association with the outcome and with predictors outside the group. It assesses the predictive value of the group as a block. It does not allocate that value among the group's members.
Conditional permutation methods attempt to shuffle a predictor while respecting its relationship with other predictors, for example by sampling within suitable strata or from an estimated conditional distribution. These methods can address some problems caused by marginal shuffling, but their results depend on how well that conditional structure is represented. They do not make a ranking automatically definitive.
Derivation: Why the Split Decrease Is Weighted
The parent node contains \(N_t\) observations. Its contribution to the tree's overall impurity, when impurity is expressed per observation in the full training set, is \((N_t/N)I(t)\). After the split, the children contribute \((N_L/N)I(L)\) and \((N_R/N)I(R)\). Subtracting the total child contribution from the parent contribution gives
This explains both the weighting and the direction of the measure: a split is useful to the tree's impurity objective when it reduces the weighted impurity. Since \(N_t=N_L+N_R\), the comparison accounts for the size of both child nodes. Summing these decreases over splits on a predictor records how much impurity reduction the fitted tree attributed to those splits. It does not establish how the same model would perform if that predictor were removed and the model refitted.
Worked Example: Reading an Unstable Ranking
Two hypothetical forests are fitted to bootstrap samples of the same training data. Their predictor correlation is 0.97, and the response is claim cost. Forest 1 reports normalized split-based shares \(A=54\%\), \(B=36\%\), and \(C=10\%\); Forest 2 reports \(A=31\%\), \(B=59\%\), and \(C=10\%\). Their held-out mean squared errors are 2.04 and 2.06. In separate held-out permutation checks, the individual importance increases are \(A=0.15\), \(B=0.10\) for Forest 1 and \(A=0.08\), \(B=0.18\) for Forest 2. Joint permutation increases are 1.7 and 1.8, respectively. These are stipulated hypothetical results, with no parametric distribution, interest rate, or mortality assumption.
Method. Compare the rankings across fits, then compare the individual permutation results with the joint results and held-out errors.
The split-based ranking reverses: Forest 1 ranks \(A\) above \(B\), while Forest 2 ranks \(B\) above \(A\). The individual permutation ranking also reverses: the increases are 0.15 versus 0.10 in Forest 1, but 0.08 versus 0.18 in Forest 2. The held-out errors differ by \(2.06-2.04=0.02\), whereas joint permutation increases are 1.7 and 1.8.
Conclusion and trap avoided. The two fits have similar predictive error, unstable individual rankings, and substantially larger joint permutation effects. Together, these results suggest that the ensemble can use the correlated pair as a useful block, while the allocation between \(A\) and \(B\) varies with the fit. The mistake is to select \(A\) or \(B\) as the definitively superior predictor from one fit's ranking. These diagnostics describe model behavior; they do not prove equal individual contributions or identify a causal driver.
Practical Interpretation and Exam Traps
When examining importance output, first identify the measure, the outcome loss or impurity, and the data set on which the calculation was made. Then inspect the relationships among predictors before interpreting individual ranks. For correlated predictors, compare rankings across reasonable refits or resamples, consider grouped permutation, and report predictive performance separately from importance. A stable prediction score can coexist with unstable variable rankings.
A frequent calculation error is to subtract losses in the wrong order. If baseline loss is 3.0 and permuted loss is 3.7, the importance is \(3.7-3.0=0.7\), not \(-0.7\). Another is to shuffle a predictor in the training data and treat the resulting loss as an unbiased evaluation: the fitted model has already used that training data. Evaluate permutation effects on data held out from fitting, and apply the same loss definition and evaluation rows to the comparisons.
- Split-based importance records accumulated impurity reduction in the fitted trees; correlated predictors may compete for splits.
- Individual permutation importance asks what happens when one predictor is disrupted while the others remain available.
- Grouped permutation importance asks what happens when a selected set is disrupted together; it does not divide group importance into unique shares.
- Neither ranking is causal. Importance depends on the model, the data, the loss or impurity measure, and the calculation procedure.
These distinctions are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. In an exam-style interpretation, explain what the score measures, note the effect of predictor correlation, and avoid claiming more than the diagnostic supports.
Check Your Understanding
For each problem, state what the calculated measure supports and avoid interpreting an importance score as a causal effect.
- A forest reports summed weighted impurity decreases of 14 for \(P\), 21 for \(Q\), and 5 for \(R\). Calculate the normalized split-based shares and identify the highest-ranked predictor. (Answer: Total \(=40\); \(P=35.0\%\), \(Q=52.5\%\), \(R=12.5\%\); \(Q\) ranks highest.)
- A model's baseline held-out mean absolute error is 3.2. Shuffling \(U\) gives an average error of 4.0, shuffling \(V\) gives 3.5, and jointly shuffling \(U,V\) gives 4.6. Calculate all three permutation increases and state which is largest. (Answer: \(U=0.8\), \(V=0.3\), joint \(=1.4\); the joint increase is largest.)
- Two strongly correlated predictors have individual permutation increases of 0.12 and 0.09, while their grouped increase is 1.25. What is the most defensible interpretation? (Answer: The pair may carry substantial shared predictive information that either predictor can partly substitute for alone; the individual values do not measure unique contributions.)
- A predictor has a large split-based score but a near-zero held-out permutation increase. Give one plausible explanation and one check. (Answer: A correlated predictor may substitute when this predictor is shuffled; inspect predictor correlations and assess a grouped permutation.)
- Across two bootstrap fits, predictor \(M\) ranks above \(N\) in one importance table and below it in the other, while held-out errors are 1.80 and 1.82 and their grouped permutation increases are 0.9 and 1.0. What should be reported? (Answer: Predictive performance is similar and the individual ranking is unstable; the pair appears useful jointly, but the results do not establish which predictor has the larger independent or causal effect.)