What Forest Variable Importance Cannot Establish

What an Importance Score Does—and Does Not—Say

A random forest can rank predictors by how much they contribute to its predictions. That ranking can help describe how a fitted model uses its inputs. It does not, by itself, explain why an outcome occurs or what would happen if a person, business, or policymaker changed one of those inputs. The distinction matters whenever a predictive analysis is used to support decisions.

For example, a hypothetical insurer might find that a property’s distance from a fire station is highly important in a forest predicting claim frequency. This means the fitted forest relies on that variable to predict outcomes in the data and evaluation setting. It does not establish that changing the property’s distance would change its claim frequency. Distance could instead act as a proxy for other characteristics, such as location or building type.

Definition: A forest variable-importance measure summarizes some aspect of a predictor’s contribution to, or use by, a fitted forest. Its interpretation depends on the particular measure, the fitted model, the other predictors, the data distribution, and—in a predictive evaluation—the loss or performance measure used. Importance is not automatically a measure of causation, unique explanatory effect, or policy value.

Two common importance approaches illustrate the distinction. Impurity-based importance aggregates the reductions in node impurity attributed to a predictor across the trees. Permutation importance measures the change in predictive performance when a predictor’s values are permuted. The first reflects how the fitted trees split; the second reflects the fitted forest’s reliance on the predictor under a specified permutation and evaluation scheme. Neither directly estimates a causal effect.

Permutation Importance as Predictive Reliance

Suppose a fitted forest $f$ is evaluated on $n$ observations $(x_i,y_i)$, using a loss function $L$, where smaller loss indicates better predictions. For predictor $j$, let $x_i^{\pi j}$ denote the same input as $x_i$, except that predictor $j$'s values have been permuted across the evaluation observations. A common permutation-importance estimate is the increase in average loss:

\widehat{I}_j = \frac{1}{n}\sum_{i=1}^{n} \left[ L\bigl(y_i,f(x_i^{\pi j})\bigr) - L\bigl(y_i,f(x_i)\bigr) \right].

This definition assumes that the fitted forest is held fixed while the evaluation inputs are altered, and that the chosen loss is appropriate for the predictive task. For squared-error regression, $L(y,\hat{y})=(y-\hat{y})^2$; for a classification task, a suitable loss might be misclassification loss or a probability-based loss. The permutation is applied to the evaluation data, not used to refit the forest in this definition.

Why the formula holds: The estimate is defined as the average loss after permutation minus the average loss before permutation. Subtracting those two averages term by term gives the displayed sum of loss differences. A positive value means the forest made worse predictions on this evaluation set after predictor $j$ was disrupted. It measures predictive reliance under this disruption—not the change in the outcome that would follow an actual intervention on $j$.

Permuting a predictor typically breaks its association with the response and with other predictors. The resulting inputs can therefore differ from plausible real-world cases. The score answers a predictive question about the fitted forest under that perturbation: how much does its performance change? It does not answer the causal question, “What would the outcome have been if this predictor had taken a different value?”

Worked Example: Calculating a Permutation Loss Increase

A hypothetical forest predicts quarterly claim counts. On four held-out policies, the observed counts are $10, 12, 14,$ and $16$, and the forest’s original predictions are $10, 12, 14,$ and $16$. After permuting one predictor, its predictions become $14, 10, 16,$ and $12$. Calculate its permutation importance using mean squared error (MSE), and state what the result establishes.

Method. Calculate the held-out MSE before and after permutation, then subtract the original MSE from the permuted MSE.

Before permutation, the four residuals (observed minus predicted) are $10-10=0$, $12-12=0$, $14-14=0$, and $16-16=0$. The sum of squared errors is $0^2+0^2+0^2+0^2=0$, so the original MSE is $0/4=0$.

After permutation, the residuals are $10-14=-4$, $12-10=2$, $14-16=-2$, and $16-12=4$. The squared errors are $16,4,4,$ and $16$, with sum $16+4+4+16=40$. The permuted MSE is $40/4=10$.

The estimated permutation importance is $10-0=10$ squared-count units of MSE. In this held-out set, disrupting the predictor worsened the forest’s predictions by 10 MSE units. This establishes predictive reliance under the stated setup; it does not show that changing the predictor would change claim counts.

Common mistake avoided: Calling the score a causal effect. It is a difference between two prediction losses, not a difference between observed outcomes under two interventions.

Importance Does Not Establish a Unique Explanatory Effect

A predictor can be important because it carries information the forest can use, even if another predictor carries nearly the same information. Conversely, a predictor can have low individual permutation importance when a correlated predictor lets the forest make similar predictions after the first variable is disrupted. Importance rankings can therefore depend on which predictors are included and how the fitted forest distributes its splits or predictions among them.

This is not the same as estimating a predictor’s unique contribution after accounting for all other information. In particular, ordinary permutation importance breaks the relationship between the permuted variable and its correlated predictors. The resulting perturbed inputs may be unusual, and the associated loss increase need not describe the predictor’s contribution among realistic observations. Conditional permutation methods can address some aspects of this problem, but they remain predictive measures; they do not turn importance into causal evidence.

Worked Example: Two Predictors Carry the Same Signal

Consider a hypothetical held-out set with response values $y=(0,0,1,1)$ and two binary predictors $A=(0,0,1,1)$ and $B=(0,0,1,1)$. They are exact duplicates. For illustration, suppose the fitted ensemble predicts $f(A,B)=(A+B)/2$, and importance is measured by the increase in MSE after permuting one predictor. Calculate the baseline MSE and the importance for each predictor when its values are permuted in the order $(1,0,1,0)$.

Method. Compute the baseline predictions, then replace one predictor at a time with its permuted values while leaving the other predictor unchanged.

Initially, the predictions are $(0+0)/2=0$, $(0+0)/2=0$, $(1+1)/2=1$, and $(1+1)/2=1$. They equal the responses, so baseline MSE is $(0^2+0^2+0^2+0^2)/4=0$.

Permuting $A$ gives $A^\pi=(1,0,1,0)$, while $B=(0,0,1,1)$. The new predictions are $0.5, 0, 1,$ and $0.5$. The errors (prediction minus response) are $0.5, 0, 0,$ and $-0.5$. Thus the permuted MSE is $(0.25+0+0+0.25)/4=0.125$, and the estimated importance for $A$ is $0.125-0=0.125$.

By symmetry, permuting $B$ alone produces the same predictions and the same importance, $0.125$. Yet neither predictor contains information unavailable from the other: either one alone exactly matches the response in this set. The two positive scores do not demonstrate two separate explanatory effects. They describe the fitted rule’s response to disrupting one input while keeping its duplicate in place.

Common mistake avoided: Treating two positive importance scores as proof that each predictor contributes independent information. Here the predictors are identical, so that conclusion cannot follow.

Importance Does Not Establish Causation or Policy Value

Causal claims concern how outcomes would differ under alternative interventions. For a binary treatment $T$, the average causal effect is commonly written $E[Y(1)-Y(0)]$, where $Y(1)$ and $Y(0)$ are the potential outcomes for the same target population under treatment and no treatment, respectively. A forest importance score is not an estimate of this contrast. Observational data can show a strong association between $T$ and $Y$ because treatment assignment is related to baseline risk, access, or other factors.

Even a genuinely predictive variable is not automatically a good policy lever. A decision requires an appropriate causal estimate of the effect of the proposed action, the costs and benefits of acting, the target population, operational constraints, and consideration of fairness and unintended consequences. Importance may help identify useful predictors for forecasting or further investigation, but it cannot replace those decision inputs.

Worked Example: A Predictive Program Flag and a Misleading Policy Conclusion

A hypothetical health organization observes a “program participant” flag in its records. Among 1,000 participants, 900 are in a low-baseline-risk group with a 1% claim rate and 100 are in a high-baseline-risk group with a 20% claim rate. Among 1,000 nonparticipants, 100 are low risk and 900 are high risk. Assume, for this illustration, that within each risk group program participation has no effect on claims. Calculate the observed rates, compare the crude association with the within-group effects, and assess a proposed program costing $40 per person when a claim costs $1,000.

Method. Weight each group’s claim rate by its share of each participation group. Then distinguish this observational comparison from the stipulated within-group causal effect.

For participants, the expected number of claims is $900(0.01)+100(0.20)=9+20=29$. Their observed rate is $29/1{,}000=0.029$, or 2.9%.

For nonparticipants, the expected number is $100(0.01)+900(0.20)=1+180=181$. Their observed rate is $181/1{,}000=0.181$, or 18.1%. The crude difference, participant minus nonparticipant, is $0.029-0.181=-0.152$, or $-15.2$ percentage points.

Within each risk group, however, the problem stipulates no program effect. The low-risk claim rate is 1% for either participation status, and the high-risk rate is 20% for either status. The average causal effect under this stipulated scenario is therefore zero; the crude difference arises from the different proportions of low- and high-risk people.

If someone incorrectly treated the 15.2-percentage-point crude difference as the program’s effect, the estimated claim savings per person would be $0.152(1{,}000)=\$152$, apparently exceeding the \$40 program cost. Under the stated no-effect assumption, expected claim savings are instead $\$0$, so the program costs \$40 per person without claim savings in this calculation. A forest might find the participation flag predictively useful because it is associated with baseline risk, but its importance could not establish that the program prevents claims or that expanding it is worthwhile.

Common mistake avoided: Reading predictive association as evidence of an intervention effect. The observed rates and the assumed within-group effects answer different questions.

How to Report an Importance Result Responsibly

A careful interpretation names the measure and its setting before making a claim. For permutation importance, state the evaluation data, loss measure, and whether a larger score means a greater loss increase. For impurity importance, describe it as importance derived from the fitted trees’ split improvements, not as a direct measure of independent information. Avoid interpreting a ranking as an effect size unless the measure genuinely estimates the quantity being discussed.

A useful report can state that a predictor was among the forest’s most important variables for prediction on a specified evaluation set. It can then identify what the result does not establish: causation, a unique contribution separate from correlated predictors, or the benefits of changing a policy. If a policy conclusion is contemplated, say what additional causal and decision evidence is needed rather than allowing the importance ranking to carry that burden.

Name the importance measure.
State whether the result is based on impurity reductions, permutation, or another defined measure.

State the predictive interpretation.
For example, say that disrupting a predictor increased held-out MSE by a stated amount under a specified evaluation procedure.

Limit the conclusion.
Do not infer a causal effect, a unique explanatory contribution, or policy value from the ranking alone.

Identify the decision evidence needed.
For policy claims, seek appropriate evidence about intervention effects, costs, target populations, and relevant constraints.

Key takeaway: Forest importance describes a feature’s role in a particular fitted model and predictive evaluation. It is useful for understanding predictive reliance, but it cannot on its own establish why outcomes occur, what would happen under an intervention, or whether a policy is worthwhile.

Check Your Understanding

For each problem, distinguish what the importance calculation supports from any stronger conclusion it cannot support.

A forest has held-out MSE 6.0 before permutation of predictor $C$ and 8.5 after permutation. What is the permutation importance in MSE units, and what does it mean? (Answer: 2.5 MSE units; the fitted forest’s held-out loss increased by 2.5 under this permutation, which does not establish a causal effect.)
Two predictors are exact copies. A forest’s ordinary permutation importance is positive for each. Does that show that each predictor contributes unique information? (Answer: No; positive scores do not establish unique information when the predictors duplicate one another.)
A program group has a 4% outcome rate and a comparison group has a 10% rate. A forest assigns high importance to program status. Can these facts alone establish that the program reduces outcomes by 6 percentage points? (Answer: No; the difference is observational, and importance is predictive. The facts alone do not identify a causal effect.)
A fitted forest’s baseline loss is 12. After independently permuting predictor $D$, the loss is 11. What is the estimated importance, and how should its sign be interpreted? (Answer: $-1$ loss unit; this permutation produced lower loss on this evaluation set, so it does not show a positive predictive reliance under this estimate.)
A policy costs \$25 per person. A predictive model assigns greater risk to people with a particular marker, and the marker has high permutation importance. Is that enough to recommend a policy targeting the marker? (Answer: No; importance alone gives neither the causal effect of the policy nor its net value after costs and constraints.)

What You'll Learn