What a Variable-Importance Ranking Says
A variable-importance chart can look like a direct answer to a business question: which factors matter most? But “importance” is not a single universal property of a variable. It is a summary produced by a particular model, importance method, data set, and—in some methods—performance metric. A ranking can help describe how a fitted model uses information for prediction. By itself, it does not show that changing a variable would change an outcome, that the variable has unique predictive value, or that acting on it would create business value.
For reporting, name the method and state what it measures. “Feature A is the most important variable” is incomplete. A more defensible statement is: “On this validation sample, permuting Feature A increased the fitted model’s mean squared error more than permuting any other single feature.” That wording identifies the model behavior being summarized without implying a causal effect.
Three common families of measures illustrate why the method must be identified:
- Permutation importance measures how much a chosen loss changes when the values of a feature are shuffled while the fitted model is held fixed.
- Tree impurity importance aggregates reductions in an impurity measure, such as Gini impurity or residual sum of squares, from splits using a feature.
- Coefficient-based summaries use fitted regression coefficients, sometimes after scaling variables. Their interpretation depends on units, coding, model form, and the other predictors in the model.
These methods do not measure the same thing. A permutation score concerns the effect of disrupting a feature on predictions under a specified loss. An impurity score describes how a fitted tree used splits. A coefficient describes a model’s fitted relationship on its chosen scale. It is therefore possible for two valid methods to rank variables differently without either calculation being erroneous.
Permutation Importance: A Change in Validation Loss
Let a fitted model produce predictions \(\hat{y}_i\) for observations in a validation sample, with observed outcomes \(y_i\). For squared-error loss, the baseline mean squared error is the average of \((y_i-\hat{y}_i)^2\). To calculate permutation importance for feature \(X_j\), shuffle that feature’s values across the validation observations, leave the fitted model unchanged, and calculate the new loss. The increase in loss is the permutation importance for that shuffle.
Here, \(\widehat{L}_{\mathrm{base}}\) is the validation loss from the original features, and \(\widehat{L}_{\mathrm{perm}(j)}\) is the loss after feature \(j\) is shuffled. A positive value means the model’s validation predictions were worse after the shuffle, under the chosen loss. A value near zero means that this shuffle produced little change in that loss. A negative value can occur from validation-sample variation: the shuffled predictions happened to score better than the baseline predictions on that sample.
Worked Example: Calculating Permutation Importance for Squared Error
A hypothetical insurer fits a model to predict repair cost, measured in thousands of dollars. On four validation claims, the observed costs are \(10, 10, 20, 20\), and the model’s baseline predictions are \(12, 12, 18, 18\). Shuffling one feature changes the predictions to \(18, 18, 12, 12\). Calculate that feature’s permutation importance using mean squared error. The method is a loss difference; the common mistake to avoid is comparing the two sets of predictions without first calculating both losses.
Baseline loss. The prediction errors are \(10-12=-2\), \(10-12=-2\), \(20-18=2\), and \(20-18=2\). Their squared errors are \(4,4,4,4\). Thus the baseline sum of squared errors is \(4+4+4+4=16\), and the baseline mean squared error is \(16/4=4\), in squared thousands of dollars.
Loss after shuffling. The errors are \(10-18=-8\), \(10-18=-8\), \(20-12=8\), and \(20-12=8\). Each squared error is \(64\), so the shuffled sum of squared errors is \(64+64+64+64=256\). The shuffled mean squared error is \(256/4=64\).
Importance and check. The estimated permutation importance is \(64-4=60\), in squared thousands of dollars. The result is positive, as expected here because the shuffled predictions are farther from every observed value than the baseline predictions. This is a statement about this fitted model’s validation loss under this shuffle; it is not an estimate of how much repair cost would change if the feature were changed.
Tree Impurity Importance: What the Fitted Splits Accumulate
A decision tree commonly chooses splits that reduce impurity within the resulting groups. For a classification tree, one possible impurity measure is Gini impurity. If a node contains a proportion \(p_k\) of observations from class \(k\), its Gini impurity is \(1-\sum_k p_k^2\). The impurity decrease at a split is the parent-node impurity minus the child-node impurities weighted by their shares of the parent observations. A tree’s impurity importance for a feature aggregates the decreases from splits that use that feature, often weighting decreases by the fraction of the full training sample reaching the split.
The formula describes how much a split improves class separation within the fitted tree. It does not measure a variable’s standalone relationship with the outcome, and it does not necessarily measure its value in a different model or population. Tree construction choices, available split points, variable type, and correlated predictors can all affect the accumulated score.
Worked Example: Aggregating Impurity Decreases in a Tree
A hypothetical classification tree is fitted to eight policies, four with an event and four without one. The root split uses Feature A, sending four policies to each child. One child contains three event cases and one non-event case; the other contains one event case and three non-event cases. The first child is then split using Feature B into a pure group of three event cases and a pure group of one non-event case. Calculate the full-sample-weighted impurity contributions of A and B. Use Gini impurity and do not normalize the contributions.
Root impurity. The event and non-event proportions are both \(4/8=0.5\). Thus the root Gini impurity is \(1-(0.5^2+0.5^2)=1-0.5=0.5\).
Contribution of A. Each child of the root has class proportions \(3/4\) and \(1/4\), in one order or the other. Each child’s Gini impurity is \(1-((3/4)^2+(1/4)^2)=1-(9/16+1/16)=6/16=0.375\). Their weighted average is \((4/8)(0.375)+(4/8)(0.375)=0.375\). The root split decrease is \(0.5-0.375=0.125\). Since A is used at the root, its full-sample-weighted contribution is \(0.125\).
Contribution of B. The node split by B has four of the eight observations, so its share of the full sample is \(4/8=0.5\). Before that split, its impurity is \(0.375\). Both children are pure, so their Gini impurities are zero and their weighted average is zero. The local decrease is \(0.375-0=0.375\), and B’s full-sample-weighted contribution is \((4/8)(0.375)=0.1875\).
Ranking and check. The unnormalized contributions are \(0.125\) for A and \(0.1875\) for B, so B has the larger impurity importance in this tree. The total decrease is \(0.125+0.1875=0.3125\), which also equals the root impurity \(0.5\) minus the weighted final impurity \(0.1875\). This checks the aggregation. It does not establish that B has greater causal or business importance.
Why Rankings Can Mislead
A variable-importance ranking is best read as a description of a particular fitted model, not as a universal ordering of the real-world drivers of an outcome. Several limitations matter when translating the numbers into a report.
- Correlated predictors can share or substitute for predictive information. A fitted model may rely heavily on one of two correlated features, even when either could serve as a useful substitute in a different fitted model. Permuting one feature can also create combinations of values that are rare or absent in the observed data.
- Importance is method-specific. Permutation importance depends on the chosen loss and validation sample. Tree impurity importance depends on the tree-building process and its split opportunities. A coefficient’s absolute size depends on measurement units and coding.
- Importance is not a causal effect. A predictive association may reflect confounding, proxy variables, selection, or other structure in the data. A ranking alone does not answer what would happen under an intervention.
- Importance is not unique business value. A high score does not establish that the variable can be changed, that an action based on it is effective, or that the action’s benefits exceed its costs and constraints.
- A ranking can hide instability. Small data or modeling changes may change close rankings. If several variables have similar values, report that they are similar rather than presenting their order as a decisive distinction.
The earlier material on variable-importance ranking disagreement is useful context: rankings can differ because models and importance definitions capture different aspects of prediction. When reporting a ranking, include enough information for a reader to understand its scope: the fitted model, the importance method, the data used, and the loss or impurity definition where relevant. Avoid implying more precision than the calculation supports.
Worked Example: Correlated Features and a Misleading Claim of Uniqueness
A hypothetical screening model is evaluated on four records. Features \(X_1\) and \(X_2\) have identical observed values, \(0,0,1,1\), and the outcomes are \(0,0,1,1\). The fitted model predicts each outcome exactly using \(X_1\) and does not use \(X_2\). For this example, shuffling \(X_1\) changes its values to \(1,0,0,1\), while the model’s prediction rule remains unchanged. Calculate baseline and permuted mean squared error, then assess whether the result proves that \(X_1\) has unique predictive value.
Baseline loss. The model predicts \(0,0,1,1\). Each prediction equals its observed outcome, so all four squared errors are zero. The baseline mean squared error is \((0+0+0+0)/4=0\).
Loss after the specified shuffle. Because the model uses \(X_1\), it predicts \(1,0,0,1\). Comparing with outcomes \(0,0,1,1\) gives errors \(1,0,-1,0\), with squared errors \(1,0,1,0\). The shuffled mean squared error is \((1+0+1+0)/4=0.5\). The permutation importance for this shuffle is \(0.5-0=0.5\).
Interpretation and check. Under this fitted model and shuffle, disrupting \(X_1\) worsens validation mean squared error by \(0.5\). But \(X_2\) has the same values and could have been used instead by another fitted model. Therefore the result does not prove that \(X_1\) has unique predictive value, much less a causal effect. The calculation is internally consistent: two of four predictions become incorrect, each with squared error \(1\), giving a mean of \(2/4=0.5\).
Scale, Reporting, and the Causal Boundary
Raw regression coefficients are especially easy to misread as a universal ranking. In a model with a linear predictor, a coefficient describes the fitted change in that predictor for a one-unit increase in a variable, holding the other model inputs fixed. A one-unit increase may mean one dollar for one variable and one year for another. Comparing absolute coefficient values without accounting for units can therefore give a meaningless ordering. Standardizing continuous predictors can make their coefficient magnitudes more comparable per standard-deviation change within that model, but it does not turn them into a universal measure of importance or solve issues such as correlated predictors and different measurement quality.
Worked Example: Why Raw Coefficient Size Is Not a Ranking
A hypothetical logistic model uses annual income measured in dollars and age measured in years. Its fitted coefficients are \(0.0004\) per dollar of income and \(0.03\) per year of age. In the validation population, the standard deviations are \(5{,}000\) dollars for income and \(8\) years for age. Compare the raw coefficient magnitudes and the fitted log-odds change for a one-standard-deviation increase in each feature. The method is a scale comparison within this fitted model; avoid calling either comparison a causal effect.
Raw magnitudes. The absolute income coefficient is \(0.0004\); the absolute age coefficient is \(0.03\). Since \(0.03>0.0004\), a naive raw-coefficient ranking would put age first. That comparison treats one dollar and one year as equivalent-sized units, which they are not.
One-standard-deviation changes. For income, the fitted log-odds change is \(0.0004\times5{,}000=2\). For age, it is \(0.03\times8=0.24\). On this particular scale-based comparison, the income change is larger: \(2>0.24\). These are changes in the model’s linear predictor, not changes in probability; the probability change depends on the starting predictor value.
Interpretation and check. The raw coefficient order and the one-standard-deviation order differ because the variables use different units. The arithmetic checks directly: the income coefficient is multiplied by \(5{,}000\), and the age coefficient by \(8\). Even the scaled comparison remains model-specific and does not establish that changing income would cause a modeled outcome to change.
A defensible report separates the calculated result from its interpretation. For example: “Permutation importance, calculated as the increase in validation mean squared error, ranked Feature A highest in this fitted model. The score describes predictive reliance under this procedure; correlated predictors may provide overlapping information, and the ranking does not establish a causal effect or the value of an intervention.” This reports the evidence, names its scope, and states what it cannot support.
A common reporting trap is to write, “Feature A is the most important driver, so changing A should reduce claims.” The first clause overstates what the ranking establishes, and the second makes a causal and decision claim unsupported by the importance calculation. A correction is: “Feature A produced the largest increase in validation loss when permuted in this fitted model. Further evidence would be needed to assess whether an intervention involving A would change claims or be operationally worthwhile.” The distinction is essential: prediction describes performance on observed data, while intervention claims require evidence about outcomes under actions.
Check Your Understanding
For each problem, identify what the importance result supports and avoid extending it beyond the stated method.
- A regression model has baseline validation mean squared error \(25\). Permuting Feature R produces mean squared error \(31\). What is the permutation importance, and what does its sign indicate? (Answer: \(31-25=6\); the positive value means loss increased under this permutation, not that changing R causes the outcome to change.)
- A tree split has parent Gini impurity \(0.48\). Two equally sized children have impurities \(0.30\) and \(0.10\). What is the impurity decrease at the split? (Answer: \(0.48-[(0.5)(0.30)+(0.5)(0.10)]=0.28\).)
- A fitted model has two identical, highly predictive input variables. Its permutation importance is large for the variable used by the model and zero for the unused one. Does this prove that the used variable has unique predictive information? (Answer: No; the unused variable may provide substitutable information, and the scores describe this fitted model’s response to the specified shuffles.)
- A linear model has coefficients \(0.5\) per unit of \(Z\) and \(0.02\) per unit of \(W\). The standard deviations are \(0.2\) for \(Z\) and \(30\) for \(W\). Which has the larger fitted linear-predictor change per standard deviation? (Answer: \(Z\) gives \(0.5(0.2)=0.10\); \(W\) gives \(0.02(30)=0.60\), so \(W\) has the larger change by this scale-based comparison.)
- A report states that a feature ranked first in tree impurity importance, “so reducing it will lower event rates.” What is the main inferential error, and what does the ranking actually summarize? (Answer: It makes an unsupported causal claim; the ranking summarizes impurity reductions from splits using that feature in the fitted tree.)