Why Correlated Predictors Complicate Importance
A random forest can predict well while giving an unstable or incomplete ranking of its predictors. This is especially likely when two or more predictors carry similar information. A tree may split on one predictor and have little reason to split on its close substitute; another tree may choose the substitute instead. As a result, importance can be divided, shifted, or masked across the group, even when the group is useful for prediction.
The practical consequence is that an individual-variable ranking is not necessarily a unique attribution of predictive value. A high-ranked member of a correlated group may be a convenient representative selected by the fitted forest, not the only useful variable. Conversely, a low individual score does not prove that the variable is irrelevant when a correlated alternative is available.
Two common forest importance measures respond differently to this overlap. Impurity importance accumulates the improvement in a split criterion attributed to each predictor across the trees. Permutation importance measures how much predictive performance worsens when the values of a predictor are shuffled for evaluation cases. The earlier material on why impurity importance favors some predictors is relevant here: impurity scores have additional selection biases, so correlation is not their only limitation.
For permutation importance, the usual forest procedure evaluates shuffled predictor values on out-of-bag cases for each tree, or uses another held-out evaluation set. A permutation breaks the link between that predictor’s values and the outcomes being evaluated. With correlated predictors, however, the shuffled values can also break the predictor’s usual relationship with its correlated partners. Thus the resulting importance is a property of the fitted model, the evaluation data, and the perturbation—not a universal measure of the predictor’s intrinsic worth.
How Correlation Redistributes Impurity Importance
Suppose two predictors \(X_1\) and \(X_2\) are strongly correlated and both help separate high-outcome cases from low-outcome cases. At a node, the tree considers candidate splits and selects one. If a split on \(X_1\) is selected, it may already make the groups sufficiently homogeneous that a similar split on \(X_2\) offers little additional improvement. That tree credits the selected improvement to \(X_1\). In another tree, sampling variation, the candidate predictors available at the node, or a tie in split quality can lead to \(X_2\) being selected instead.
The following calculation illustrates the accounting, not a claim that a particular forest must produce these values. Suppose a forest’s total impurity reductions, summed over its trees, are 40 units. Of these, 18 are credited to \(X_1\), 15 to \(X_2\), and 7 to other predictors. The individual shares are \(18/40=45\%\), \(15/40=37.5\%\), and \(7/40=17.5\%\). Considered as a group, \(X_1\) and \(X_2\) account for \(33/40=82.5\%\) of the total.
A different forest fit might allocate 21 units to \(X_1\) and 12 to \(X_2\), or 13 to \(X_1\) and 20 to \(X_2\), while keeping the combined total at 33. The individual ranking changes, but the group’s share remains 82.5%. Such a pattern is plausible when the predictors act as substitutes at split locations. It does not imply that the two predictors have identical effects or that their individual contributions can always be recovered by adding their scores.
Worked Example: Reading an Impurity Ranking as a Group
A hypothetical regression forest has a total of 50 units of accumulated impurity reduction. Two correlated predictors, \(X_1\) and \(X_2\), receive 17 and 13 units, respectively. All other predictors receive 20 units. A second fit with a different random seed assigns 12 units to \(X_1\) and 18 to \(X_2\), while the other predictors still receive 20. Compare the rankings and interpret the group.
Method. Calculate each predictor’s share of the total in each fit, then calculate the combined share for the correlated pair.
First fit. The shares are \(17/50=34\%\) for \(X_1\), \(13/50=26\%\) for \(X_2\), and \(20/50=40\%\) for all other predictors combined. The pair’s total is \(17+13=30\) units, so its combined share is \(30/50=60\%\).
Second fit. The shares are \(12/50=24\%\) for \(X_1\), \(18/50=36\%\) for \(X_2\), and \(20/50=40\%\) for the other predictors. The pair still has \(12+18=30\) units, or \(30/50=60\%\).
Conclusion. The first fit ranks \(X_1\) above \(X_2\); the second ranks \(X_2\) above \(X_1\). Yet the combined share is 60% in both. The group-level description is more stable in this example. It would be a mistake to claim that the first fit establishes \(X_1\) as uniquely more important.
Common mistake avoided: Treating the top individual impurity score as the exclusive source of the group’s predictive signal. The calculation shows that the total credited to correlated predictors can be more informative than the order within the pair.
Why Permutation Importance Can Mask a Predictor
With correlated predictors, shuffling one variable may leave much of the forest’s predictive ability intact because its correlated partner still carries similar information. This can make the shuffled variable’s individual permutation importance small. The forest is not retrained after the shuffle: the calculation asks how the already-fitted model performs when evaluation values are perturbed. It does not ask how a new forest would adapt after losing a predictor.
The increase-in-loss definition follows directly from the comparison being made. Let \(L_{\mathrm{base}}\) be the mean loss on the unaltered evaluation cases, and \(L_{j,\mathrm{perm}}\) the mean loss on those same cases after shuffling predictor \(j\). The difference isolates the change in loss under that perturbation:
The losses must be calculated on the same cases and with the same loss function for the difference to have this interpretation. For regression, one possible choice is mean squared error. For classification, one may use a classification loss such as misclassification loss or log loss, provided the same definition is used before and after permutation. A decrease in loss gives a negative value; that can occur through evaluation variation and should not automatically be interpreted as evidence that the predictor harms the model.
Worked Example: Calculating Individual Permutation Importance
A hypothetical classification forest is evaluated on four out-of-bag cases. The outcome is coded as 0 or 10, and for this illustration the evaluation loss is squared error between the numeric outcome and the forest prediction. The forest uses two strongly correlated predictors, \(X_1\) and \(X_2\). Calculate the permutation importance for each predictor from the supplied predictions.
| Case | Outcome | Baseline prediction | Prediction after permuting \(X_1\) | Prediction after permuting \(X_2\) |
|---|---|---|---|---|
| 1 | 0 | 1 | 1 | 1 |
| 2 | 0 | 1 | 9 | 1 |
| 3 | 10 | 9 | 1 | 9 |
| 4 | 10 | 9 | 9 | 9 |
Method. Compute mean squared error for the baseline and each set of permuted predictions. Subtract the baseline mean squared error from each permuted mean squared error.
Baseline loss. The four squared errors are \((0-1)^2=1\), \((0-1)^2=1\), \((10-9)^2=1\), and \((10-9)^2=1\). Their sum is \(1+1+1+1=4\), so the baseline mean squared error is \(4/4=1\).
After permuting \(X_1\). The squared errors are \((0-1)^2=1\), \((0-9)^2=81\), \((10-1)^2=81\), and \((10-9)^2=1\). Their sum is \(1+81+81+1=164\), so the permuted mean squared error is \(164/4=41\). Therefore, \(I_1=41-1=40\).
After permuting \(X_2\). The predictions match the baseline predictions in this supplied evaluation, so the squared errors are \(1,1,1,1\). The permuted mean squared error is \(4/4=1\), and \(I_2=1-1=0\).
Conclusion. This fitted forest’s predictions are sensitive to shuffling \(X_1\), but not to the supplied shuffle of \(X_2\). A zero score for \(X_2\) does not establish that the underlying information represented by \(X_2\) is useless: in this example the forest can retain the relevant signal through its use of \(X_1\). The scores describe these fitted predictions under these perturbations.
Common mistake avoided: Calling \(I_2=0\) proof that \(X_2\) has no relationship with the outcome. The calculation only shows no loss increase for this particular fitted forest and permutation.
Grouped Interpretation
When predictors are correlated, the question may concern the information carried by the set rather than which member receives the largest individual score. A group permutation can address that question: apply the same row permutation to every predictor in the group, so their values remain paired with one another, and then measure the change in evaluation loss. This breaks the group’s alignment with the outcomes while retaining the observed relationships among the group members.
Group importance is not generally the sum of individual permutation importances. Each individual shuffle leaves other correlated predictors in place, while a group shuffle perturbs all of them together. Because the fitted forest can use overlapping information, the loss increases from the separate shuffles may be much smaller than the increase from shuffling the group. Conversely, individual effects and joint effects can differ in other ways when the forest uses interactions.
Worked Example: Measuring the Combined Contribution
A hypothetical regression forest is evaluated on four cases using squared error. Two correlated predictors are considered together. The baseline predictions and predictions after both predictors are jointly permuted are shown below. Calculate the group permutation importance and explain why it cannot be inferred by summing individual scores without further calculation.
| Case | Outcome | Baseline prediction | Prediction after jointly permuting the group |
|---|---|---|---|
| 1 | 2 | 2.5 | 1.5 |
| 2 | 2 | 1.5 | 8.5 |
| 3 | 8 | 7.5 | 2.5 |
| 4 | 8 | 8.5 | 7.5 |
Baseline loss. The squared errors are \((2-2.5)^2=0.25\), \((2-1.5)^2=0.25\), \((8-7.5)^2=0.25\), and \((8-8.5)^2=0.25\). The mean is \((0.25+0.25+0.25+0.25)/4=1/4=0.25\).
Jointly permuted loss. The squared errors are \((2-1.5)^2=0.25\), \((2-8.5)^2=42.25\), \((8-2.5)^2=30.25\), and \((8-7.5)^2=0.25\). The sum is \(0.25+42.25+30.25+0.25=73\), so the mean is \(73/4=18.25\).
Group importance. The increase in mean squared error is \(18.25-0.25=18.00\). This is the measured importance of the jointly perturbed group for this fitted forest and evaluation set. Individual importance calculations would require their own predictions after each separate shuffle; the group result cannot be obtained by simply assigning it to one predictor or assuming it equals a sum of individual scores.
Common mistake avoided: Treating a group score as a unique split of credit among its members. This calculation measures the combined loss increase; it does not determine a uniquely correct allocation of that increase to either predictor.
Reading Rankings With Appropriate Caution
A defensible interpretation separates three questions:
Inspect the data and identify strongly related predictors, including variables that represent overlapping measurements or derived versions of similar information.
State whether the ranking comes from accumulated impurity reduction or an increase in evaluation loss after permutation. These measures answer different questions.
When predictors overlap, report the individual scores with the correlation context, and consider a group-level evaluation if the joint information is the question.
If the intended conclusion is about the unique contribution of one predictor after accounting for its correlated partners, an ordinary individual permutation score is not sufficient. That question requires a method designed to preserve or otherwise account for predictor relationships. For this tutorial’s purpose, the key point is not to treat ordinary rankings as unique attribution.
Importance is also not a causal measure. A forest ranking does not show that changing a predictor would change the outcome, and correlation among predictors can make even predictive attribution ambiguous. Report the fitted model, the importance definition, the evaluation basis, and the presence of correlated predictors. When the individual order shifts but the group contribution is similar, that distinction is more useful than presenting a precise-looking rank without qualification.
Check Your Understanding
For each problem, distinguish the numerical calculation from the interpretation it supports.
- A forest has 40 total impurity-reduction units. Correlated predictors \(A\) and \(B\) receive 14 and 10 units. What percentage of total importance is assigned to the pair? (Answer: \(24/40=60\%\).)
- On five evaluation cases, a regression forest has baseline mean squared error 3.2. After permuting predictor \(C\), the mean squared error is 8.7. Calculate the permutation importance as an increase in loss. (Answer: \(8.7-3.2=5.5\).)
- A predictor’s permutation importance is \(-0.4\) under a chosen evaluation loss. State what the sign means and one conclusion it does not justify. (Answer: The observed permuted loss was 0.4 lower than baseline; this does not establish that the predictor is harmful or causally protective.)
- Two correlated predictors have individual permutation importance increases of 0.6 and 0.8. After jointly permuting the pair, loss increases by 4.1. What is the group importance, and why is it not the sum of the individual values? (Answer: Group importance is 4.1; joint perturbation changes both predictors together, so it is a different calculation from the two separate shuffles.)
- Across two forest fits, a correlated pair’s impurity scores change from \((19,11)\) to \((12,18)\), while the total importance remains 50 in each fit. What stays constant for the pair, and what changes? (Answer: The pair remains at 30 units, or 60% of total importance; the higher-ranked individual changes from the first predictor to the second.)