Correlated Predictors And Variable Importance

Why Correlated Predictors Complicate Importance

A forest may predict well using several predictors that contain much of the same information. For example, a hypothetical insurance analysis might include both a property’s floor area and its number of rooms. If those measurements are strongly related, permuting one may leave the other available as a substitute. The resulting importance of either variable can be modest even when the information represented by the pair is valuable.

The interpretation depends on the question being asked. Marginal permutation importance asks how much the fitted model’s loss increases when one predictor is disrupted, leaving the others as observed. Grouped permutation importance asks how much loss increases when a set of predictors is disrupted together. A conditional approach instead asks how much information a predictor contributes beyond what can be predicted from the others.

Definition: For a fitted model $f$, a loss function $L$, and predictor $X_j$, marginal permutation importance is the expected increase in loss when $X_j$ is replaced by a randomly permuted version $X_j'$, while the other predictors are left unchanged: $$ I_j = E\left[L\{Y,f(X_1,\ldots,X_j',\ldots,X_p)\}\right] - E\left[L\{Y,f(X_1,\ldots,X_j,\ldots,X_p)\}\right]. $$ In a forest, this increase is commonly estimated using out-of-bag (OOB) observations. For a group of predictors, permute their values together using the same row permutation, so their dependence within the group is retained.

The expectations describe population performance; practical estimates use a finite evaluation sample. For regression with squared-error loss, $L(y,\hat y)=(y-\hat y)^2$. For classification, the chosen loss might instead be misclassification error or a probability-based loss. Importance values therefore depend on the model, evaluation data, permutation procedure, and loss measure.

Marginal Importance Can Be Shared or Masked

When a predictor is permuted, its replacement is usually independent of the original row’s response and predictors. If predictors are correlated, this can create predictor combinations that are unusual in the observed data. Also, an unpermuted correlated predictor may still carry much of the disrupted predictor’s signal. Thus, a small individual importance does not necessarily mean that the information shared by a correlated group is unhelpful.

The following calculation shows both the effect of correlation and why two marginal importance values need not be equal, even when the fitted prediction treats the predictors symmetrically. The response determines which predictor is more closely tied to the outcome.

Worked Example: Unequal Individual Importance With Correlated Predictors

Assume $X_1$ and $X_2$ are jointly normal, each with mean 0 and variance 1, and with correlation 0.8. The response is $Y=X_1$, and a fitted regression forest is represented for this calculation by $f(X_1,X_2)=(X_1+X_2)/2$. Use squared-error loss and population mean squared error (MSE). A permuted value is an independent draw from the predictor’s marginal distribution.

Baseline loss. The prediction error is $\{(X_1+X_2)/2\}-X_1=(X_2-X_1)/2$. Since $\operatorname{Var}(X_2-X_1)=1+1-2(0.8)=0.4$,

L_{\mathrm{base}}=\frac{0.4}{4}=0.1.

Permute $X_2$. Let $X_2'$ be independent of $X_1$. The error becomes $(X_2'-X_1)/2$. Its MSE is $(1+1)/4=0.5$, so the importance is $0.5-0.1=0.4$.

Permute $X_1$. Let $X_1'$ be independent of the original predictors. The error is $(X_1'+X_2)/2-X_1=X_1'/2+X_2/2-X_1$. The variance contributions are $1/4$, $1/4$, and $1$. The covariance contribution between $X_2/2$ and $-X_1$ is $2(1/2)(-1)(0.8)=-0.8$. The other covariance contributions are zero because $X_1'$ is independent. Therefore,

L_{\mathrm{perm},1} = \frac14+\frac14+1-0.8 = 0.7, \qquad I_1=0.7-0.1=0.6.

The results are $I_1=0.6$ and $I_2=0.4$. The fitted prediction gives equal coefficients to the predictors, but $Y=X_1$, not $X_2$. Permuting $X_1$ therefore disrupts information more closely tied to the response. The calculations check: both permuted MSEs exceed the baseline MSE of 0.1.

This example also illustrates why symmetry of a fitted formula does not imply symmetry of its permutation importances: importance measures loss relative to $Y$, not just the change to the prediction formula.

Deriving the Loss Increase

For squared-error loss, the change in importance can be understood by comparing the error before and after permutation. Under the model and distributional assumptions in the preceding example, the original error is $(X_2-X_1)/2$. Its variance is determined by the covariance of the original predictors. After marginally permuting $X_2$, the replacement $X_2'$ is independent of $X_1$, so the covariance term disappears.

Compute the original error variance.
With unit variances and correlation $\rho$, $\operatorname{Var}\{(X_2-X_1)/2\}=(1-\rho)/2$.

Compute the permuted error variance.
Because $X_2'$ is independent of $X_1$, $\operatorname{Var}\{(X_2'-X_1)/2\}=(1+1)/4=1/2$.

Subtract the baseline.
The marginal permutation importance for $X_2$ is $1/2-(1-\rho)/2=\rho/2$.

This derivation relies on the stated zero means, unit variances, correlation $\rho$, response $Y=X_1$, prediction $f=(X_1+X_2)/2$, independent marginal replacement, and squared-error loss. It is not a general formula for forest importance. It shows specifically how breaking the original covariance changes the loss.

Grouped Importance Measures Shared Predictive Value

For a correlated group, the predictors should be permuted together using the same row permutation. This preserves their within-group relationship while disconnecting the group from the original response and the predictors left in place. Group importance is not generally the sum of individual importances: individual permutations can leave substitutes available, whereas a group permutation removes the group’s signal as a whole.

Worked Example: Individual Versus Grouped Importance

Suppose $Z,e_1,e_2$ are independent, have mean 0, and have variances $1,0.25,0.25$, respectively. Let $X_1=Z+e_1$, $X_2=Z+e_2$, and $Y=Z$. Use the fixed fitted prediction $f=(X_1+X_2)/2$ and squared-error loss. For marginal permutation, replace a single predictor with an independent draw from its marginal distribution. For grouped permutation, take $(X_1,X_2)$ together from another independent observation.

Baseline. The prediction error is $(e_1+e_2)/2$. Independence gives $\operatorname{Var}(e_1+e_2)=0.25+0.25=0.5$, so baseline MSE is $0.5/4=0.125$.

Permute $X_1$ alone. The new error is $(X_1'+X_2)/2-Z=(X_1'-Z+e_2)/2$. The three terms in the numerator are independent, with variances $1.25$, $1$, and $0.25$. Thus,

L_{\mathrm{perm},1}=\frac{1.25+1+0.25}{4}=0.625, \qquad I_1=0.625-0.125=0.5.

By the same calculation, permuting $X_2$ alone also gives MSE $0.625$ and importance $0.5$. Each remaining predictor still carries information about $Z$.

Permute the pair together. The replacement pair has the same distribution as the original pair and is independent of the original $Z$. Each replacement predictor has variance $1.25$, and their covariance is $\operatorname{Var}(Z)=1$. Therefore, the variance of their average is $(1.25+1.25+2)/4=1.125$. The average is independent of the original response $Z$, whose variance is 1, so the permuted MSE is $1.125+1=2.125$. Group importance is $2.125-0.125=2.0$.

The individual importances total 1.0, while group importance is 2.0. The difference is not an arithmetic inconsistency: individual permutations leave a correlated substitute in the prediction, but grouped permutation disrupts both predictors. All three permuted losses exceed the baseline MSE of 0.125.

In an actual forest, the pattern may differ. Trees may select one correlated predictor more often, split the signal across predictors, or use different substitutes in different trees. Importance is consequently a description of how this fitted model uses available information under a specified perturbation, not a unique allocation of the underlying signal to variables.

Conditional Permutation and Unique Information

Marginal permutation breaks a predictor’s relationship with all the other predictors. Conditional permutation instead replaces a predictor with a draw from its distribution given the remaining predictors. This aims to preserve the predictor relationships and assess whether the variable adds predictive information beyond those remaining. The conditional distribution must be specified or estimated; results can depend on that choice.

Worked Example: Marginal and Conditional Importance Answer Different Questions

Let $X_1$ and $X_2$ be jointly normal with means 0, variances 1, and correlation 0.8. Suppose $Y=X_1$, the fitted prediction is $f=(X_1+X_2)/2$, and the loss is squared error. Compare marginal permutation of $X_2$ with conditional replacement from the exact distribution of $X_2$ given $X_1$.

The baseline MSE is $(1-0.8)/2=0.1$. Marginal permutation gives $X_2'$ independent of $X_1$; as calculated directly from the error $(X_2'-X_1)/2$, its MSE is $0.5$, for marginal importance $0.5-0.1=0.4$.

For conditional replacement, joint normality gives $X_2^*\mid X_1\sim N(0.8X_1,0.36)$. Equivalently, write $X_2^*=0.8X_1+U$, where $U$ has mean 0 and variance $0.36$ and is independent of $X_1$. The replacement error is \[ \frac{X_1+X_2^*}{2}-X_1 = \frac{-0.2X_1+U}{2}. \] Its MSE is $\{0.04(1)+0.36\}/4=0.1$. Conditional importance is therefore $0.1-0.1=0$.

The result is consistent with the data-generating relationship $Y=X_1$: once $X_1$ is known, $X_2$ adds no further information about $Y$. Marginal permutation reports a positive increase because it breaks the $X_2$-$X_1$ relationship; conditional replacement does not. This example assumes the conditional distribution is known exactly and does not imply that conditional importance is always easy to estimate.

Interpretation and Common Traps

Do not treat small individual importance as proof that a predictor is useless. A correlated predictor may act as a substitute. Check whether a scientifically or operationally meaningful group has substantial joint importance.
Do not add individual importances and call the result group importance. The perturbations are different. Individual permutations leave correlated predictors intact; a grouped permutation disrupts them together.
Do not assume correlated predictors must have equal importance. Their importance depends on their relationship to the response and on how the fitted forest uses them.
Do not interpret permutation importance causally. It measures a change in predictive loss under a specified disruption. It does not estimate the effect of intervening on a predictor.
Do not confuse marginal and conditional questions. Marginal importance asks what happens when the observed predictor relationship is broken. Conditional importance asks about information beyond what the other predictors already provide.

Key takeaway: With correlated predictors, importance is an allocation problem as well as a prediction-performance measure. Report the perturbation and loss, interpret individual values alongside relevant group values, and state whether the goal is marginal or unique predictive contribution. These distinctions are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each problem, identify the type of importance being calculated and check whether the stated replacement preserves dependence within a predictor group.

Let $X_1,X_2$ be jointly normal with means 0, variances 1, and correlation 0.6. Let $Y=X_1$, $f=(X_1+X_2)/2$, and use squared-error loss. Find baseline MSE, then the MSE and importance for separately permuting $X_1$ and $X_2$. (Answer: Baseline MSE $0.2$; permuting $X_1$, MSE $0.9$ and importance $0.7$; permuting $X_2$, MSE $0.5$ and importance $0.3$.)
Suppose $X_1=X_2=Z$, where $Z$ has mean 0 and variance 1; $Y=Z$, and $f=(X_1+X_2)/2$. Under squared-error loss, find baseline MSE, either single-variable marginal importance, and grouped importance when both predictors are permuted together. (Answer: Baseline MSE $0$; each individual importance $0.5$; grouped importance $2$.)
Let $Z,e_1,e_2$ be independent with mean 0 and variances $1,0.5,0.5$. Set $X_1=Z+e_1$, $X_2=Z+e_2$, $Y=Z$, and $f=(X_1+X_2)/2$. Find baseline MSE and single-variable marginal importance. (Answer: Baseline MSE $0.25$; each individual marginal importance $0.5$.)
Let $X_1,X_2$ be jointly normal with unit variances, means 0, and correlation 0.5. With $Y=X_1$ and $f=(X_1+X_2)/2$, compare marginal and exact conditional permutation importance for $X_2$, using squared-error loss. (Answer: Marginal importance $0.25$; conditional importance $0$.)
An OOB evaluation gives baseline MSE 8. Permuting predictor A alone gives MSE 11, permuting B alone gives MSE 10, and permuting the correlated pair (A, B) together gives MSE 15. Find the three increases and explain what the group result says that the individual results do not. (Answer: A increases MSE by 3, B by 2, and the group by 7; the pair together carries predictive information that individual permutations can partly mask because the other correlated predictor remains available.)

What You'll Learn