What Feature Importance Can—and Cannot—Say
A feature-importance chart can help a reader understand which inputs a fitted model relies on when producing predictions. It does not, by itself, explain why an outcome occurred or what would happen if someone changed a predictor. That distinction matters in actuarial work: a model may use a feature to predict claims or lapses, but the resulting ranking is not evidence that changing the feature would change the outcome.
Importance is not one universal quantity. It is a family of summaries, each defined by a particular model, data set, and calculation. A report should therefore name the importance method, state the data on which it was evaluated, and describe the result as a property of the model’s predictive behavior.
The phrase “important feature” is incomplete unless the report makes clear what “important” means. A large coefficient, a large decrease in tree impurity, and a large increase in validation loss after permutation are different quantities. They need not rank features in the same order.
Permutation Importance: A Loss-Based Definition
Permutation importance asks how much worse a fixed model predicts when the values of one feature are disrupted. The calculation is made on a specified evaluation data set. For a loss where smaller values indicate better predictions, the importance of feature \(j\) is the loss after permutation minus the original loss.
The loss must be appropriate to the prediction task. For example, classification may use log loss, while numeric prediction may use squared error or absolute error. The chosen loss affects both the size and interpretation of an importance value. Permutation importance can be negative: randomizing a feature may happen to improve the measured loss, particularly with a small evaluation sample or an unhelpful feature. That result does not establish that the feature is harmful in every setting.
Worked Example: Ranking Two Features by Permutation Log Loss
A hypothetical insurer evaluates a binary claim-alert model on four held-out records. For each record, the model assigns probability \(0.90\) to the class that actually occurred. The original mean log loss is therefore calculated from those four observed-class probabilities. After separately permuting feature A, each record’s probability for its observed class is \(0.60\); after separately permuting feature B, each is \(0.75\). Calculate and compare the two permutation importances. Use natural logarithms and round final importance values to four decimal places.
Method. For a record whose observed class has predicted probability \(q\), the log loss is \(-\ln(q)\). The mean of four identical losses is that same loss. For each feature, subtract the original mean loss from the mean loss after its permutation.
Original loss. Each of the four records has loss \(-\ln(0.90)=0.1053605\ldots\). Thus the original mean is $$ L = \frac{4[-\ln(0.90)]}{4} = -\ln(0.90) = 0.1053605\ldots . $$
Feature A. The permuted mean loss is $$ L_{\pi(A)} = \frac{4[-\ln(0.60)]}{4} = -\ln(0.60) = 0.5108256\ldots . $$ So its importance, calculated before rounding, is $$ I_A^{\mathrm{perm}} = -\ln(0.60)-[-\ln(0.90)] = \ln(0.90/0.60) = \ln(1.5) = 0.4054651\ldots . $$ Rounded to four decimal places, \(I_A^{\mathrm{perm}}=0.4055\).
Feature B. Its permuted mean loss is $$ L_{\pi(B)} = \frac{4[-\ln(0.75)]}{4} = -\ln(0.75) = 0.2876821\ldots . $$ Therefore, $$ I_B^{\mathrm{perm}} = -\ln(0.75)-[-\ln(0.90)] = \ln(0.90/0.75) = \ln(1.2) = 0.1823216\ldots . $$ Rounded to four decimal places, \(I_B^{\mathrm{perm}}=0.1823\).
Conclusion and check. Both permutations increase log loss, and the increase is larger for feature A: \(0.4055>0.1823\). On these records, under this model and this measure, the prediction loss is more sensitive to disrupting A. This does not mean A causes the outcome. As a check, each importance is positive because each permuted loss exceeds \(0.1053605\ldots\). The common mistake is subtracting separately rounded losses; the calculation above subtracts unrounded values and rounds only the final result.
Why a Permutation Loss Difference Measures Model Reliance
The justification follows directly from the definition of predictive loss. The fitted model and the evaluation outcomes are held fixed. Permuting one feature changes the inputs supplied to the model, and the resulting loss records how prediction quality changes under that disruption.
Use the same records, outcomes, fitted model, and loss for the original and permuted calculations. Otherwise, the difference could reflect more than the feature disruption.
Calculate \(L\), the mean loss using the evaluation data in its original form.
Permute its values across records and calculate \(L_{\pi(j)}\), without refitting the model.
\(L_{\pi(j)}-L\) isolates the change in measured loss associated with this perturbation, under the stated setup. It does not isolate a causal effect.
The result depends on the permutation scheme. Ordinary permutation breaks the association between that feature and the outcome and also breaks its association with other features. If two predictors carry overlapping information, disrupting one may have limited effect because the model can still use the other. Conversely, an unusual combination of feature values created by permutation may be unlike any record in the original data. These limitations are reasons to explain the method, not reasons to interpret the number causally.
Tree-Based Importance and Aggregated Split Contributions
A tree can be summarized by how much its splits reduce an impurity measure, such as Gini impurity for classification or squared-error impurity for numeric prediction. A common tree-based importance aggregates the reductions attributed to each feature over the tree, often with reductions weighted by the number or proportion of records reaching the relevant node. Ensembles may aggregate these contributions across trees.
This measure describes how the fitted trees used features to make splits. It is not the same as the change in held-out predictive loss when a feature is disrupted. Its value can be affected by the available split points, the feature’s scale or number of possible categories, the tree-building procedure, and the other features offered to the model. A normalized chart may show shares that sum to 100%, but those shares are relative to the features and contributions included in that model.
Worked Example: Interpreting Normalized Tree Importance
A hypothetical random forest for predicting equipment service events reports the following total weighted decreases in node impurity: operating hours, \(0.24\); equipment age, \(0.12\); and service region, \(0.04\). These are unnormalized contributions from the same fitted model. Calculate each feature’s share of their total and state what the ranking supports.
Method. Add the reported contributions, then divide each feature’s contribution by that total. This converts the displayed values to relative shares; it does not change the underlying model.
The total contribution is \(0.24+0.12+0.04=0.40\). Operating hours account for \(0.24/0.40=0.60\), or 60%. Equipment age accounts for \(0.12/0.40=0.30\), or 30%. Service region accounts for \(0.04/0.40=0.10\), or 10%. The shares sum to \(60\%+30\%+10\%=100\%\), which checks the normalization.
Conclusion and check. In this fitted forest, operating hours have the largest share of the reported weighted split contribution, followed by equipment age and service region. The calculation does not show that changing operating hours would change service-event risk, nor that the same ranking would appear in another model or data set. The common mistake is to describe the 60% share as “60% of service events are caused by operating hours.” It is only 60% of these three features’ summed split contributions.
Different Methods Answer Different Questions
A coefficient-based summary, a tree-based split summary, permutation importance, and feature ablation are not interchangeable. A coefficient’s magnitude depends on the feature’s scale and model parameterization. Tree contributions summarize splits used during fitting. Permutation importance measures loss change when values are disrupted for a fitted model. Ablation compares predictive performance for a model trained without a feature against a model trained with it. Each method requires a clear description of its comparison.
| Method | What it summarizes | Important qualification |
|---|---|---|
| Permutation importance | Change in a specified loss after permuting a feature for a fitted model | Depends on loss, evaluation data, and permutation scheme |
| Tree split contribution | Aggregated impurity reduction attributed to a feature’s splits | Reflects the fitted tree-building procedure and available split choices |
| Feature ablation | Performance difference between models trained with and without a feature | Depends on refitting, tuning, and keeping the evaluation comparison fair |
Worked Example: Reporting an Ablation Comparison
A hypothetical model predicts monthly repair cost in dollars. On the same held-out records, a tuned model using all candidate inputs has RMSE of \(8.0\) dollars. A second model, refitted and tuned without the “days since last inspection” feature, has RMSE of \(8.6\) dollars. Calculate the absolute and relative increase and write a defensible interpretation.
Method. Feature ablation compares two models’ performance on the same evaluation records. The absolute increase is the reduced model’s RMSE minus the full model’s RMSE: $$ 8.6-8.0=0.6 \text{ dollars}. $$ The relative increase, using the full model’s RMSE as the denominator, is $$ \frac{8.6-8.0}{8.0}=\frac{0.6}{8.0}=0.075=7.5\%. $$
Conclusion and check. On these held-out records, the model refitted without the inspection-timing feature has RMSE \(0.6\) dollars higher, a \(7.5\%\) increase relative to the full model’s RMSE. This supports saying that including the feature was useful for predictive performance in this comparison. It does not establish that changing inspection timing would reduce repair cost. The common mistake is to call \(0.6\) dollars the causal effect of inspection timing; it is a difference between two models’ prediction errors.
Dependence, Stability, and Causal Language
Importance can change when the fitted model changes, even if the underlying data are similar. Different algorithms can represent relationships differently; different tuning choices can produce different fitted models; and different loss functions emphasize different prediction errors. A ranking is therefore not a permanent property of a feature.
Correlated predictors deserve particular care. If two variables contain overlapping predictive information, a model may split its use between them, select one in preference to the other, or use one as a substitute when the other is unavailable. As a result, the individual importance assigned to either feature may be smaller, unstable, or dependent on the chosen method. Ordinary permutation can also create combinations of predictor values that are rare or absent in the evaluation data, making the loss change difficult to interpret as a realistic scenario.
None of these predictive importance measures answers a causal question. A causal claim concerns what would happen to an outcome under an intervention that changes a feature, and requires an appropriate design and assumptions. A feature may be predictive because it is associated with an underlying cause, acts as a proxy, or records information collected after an event. Importance alone does not distinguish these explanations. Even a very large importance is not evidence that intervening on the feature will change the outcome.
Common Reporting Traps
- Calling importance causation. “The model relied more on feature A under this method” is a predictive statement. “Feature A caused the outcome” is a causal claim and is not established by an importance chart.
- Leaving the method unspecified. “Feature A is most important” omits whether importance means coefficient magnitude, split contribution, permutation loss change, or another measure.
- Comparing incompatible values. Values from different models, target definitions, loss functions, or data sets may not be comparable even if their labels look alike.
- Hiding dependence among features. With correlated predictors, an individual ranking can understate shared information or change when features are substituted or removed.
- Rounding too early. Calculate differences and ratios using unrounded values, then round the reported result. Rounded intermediate values can alter the final displayed answer.
A concise report can still be precise: “On the held-out evaluation set, permuting operating hours increased mean log loss by 0.12, the largest increase among the evaluated features. This indicates that this fitted model’s predictions were most sensitive to that perturbation under the selected loss; it does not estimate the effect of changing operating hours.” This wording states the method, setting, comparison, and limit of inference.
Check Your Understanding
For each problem, distinguish the numerical result from what it permits you to conclude.
- On a held-out binary classification set, every observed-class probability gives original log loss \(-\ln(0.85)\). After permuting feature M, every observed-class probability is \(0.70\). Calculate the permutation importance as an unrounded log ratio, then round to four decimals. (Answer: \(\ln(0.85/0.70)=0.1942\); loss increased, indicating model reliance under this setup, not causation.)
- A tree model has weighted split contributions of \(0.21\) for feature P, \(0.09\) for feature Q, and \(0.06\) for feature R. Find each normalized share and check the total. (Answer: Total \(0.36\); P \(58.33\%\), Q \(25.00\%\), R \(16.67\%\); shares total \(100\%\) after rounding.)
- For the same held-out numeric-prediction records, the full model has RMSE \(12\), and a model refitted without feature S has RMSE \(15\). Find the absolute and relative increase using the full model as the reference. (Answer: RMSE increases by \(3\) units, or \(3/12=25\%\); this is a predictive model comparison, not an effect of intervening on S.)
- Two strongly correlated features each have modest permutation importance when permuted separately. Is it valid to conclude that the information they contain is unhelpful? (Answer: No; either feature may partly substitute for the other, so individual permutation importance can be modest despite useful shared predictive information.)
- A report says, “The high importance of prior inspection frequency proves that increasing inspections will lower future repair costs.” Identify the unsupported inference and give a defensible replacement. (Answer: Importance does not establish a causal effect or the direction of an intervention; say that the fitted model relied on inspection frequency for prediction under the stated importance method and evaluation data.)