Why a Single Ranking Can Be Misleading
A model comparison may produce several plausible-looking lists of “important” predictors: the largest absolute GLM coefficients, the variables retained by a lasso fit, and the variables with the highest tree-based importance scores. These lists do not necessarily answer the same question. A coefficient describes a model effect on a specified scale; a lasso selection describes the result of a penalized fitting procedure; and a tree importance score summarizes a chosen measure of split contribution or predictive degradation.
Differences between these rankings are not automatically evidence that one model is wrong. They can arise from predictor units, correlation, functional form, the fitting penalty, or the definition of importance. Comparisons are most useful when the analyst first identifies what each quantity measures and what assumptions are behind it.
What the Different Rankings Measure
For a GLM with linear predictor \(\eta_i=\beta_0+\sum_j\beta_jx_{ij}\), the coefficient \(\beta_j\) is the change in the linear predictor associated with a one-unit increase in \(x_j\), holding the other included predictors fixed. The scale of \(\eta_i\) depends on the link. For example, in logistic regression, a one-unit increase changes the log odds by \(\beta_j\), and multiplies the odds by \(e^{\beta_j}\). The coefficient is conditional on the model’s other terms; it is not, by itself, a measure of out-of-sample predictive value or causal effect.
A lasso fit minimizes a loss plus a penalty on coefficient magnitudes. In the squared-error case, a common objective is:
A selected variable has a nonzero fitted coefficient at the chosen penalty value \(\lambda\). Selection answers which variables the penalized fit uses under that specification, not which variables have the greatest intrinsic importance. With correlated predictors, the lasso may select one representative, several representatives, or different representatives under small data or tuning changes.
Tree-based importance also has multiple definitions. For a tree, impurity importance adds the weighted impurity reductions from splits using a predictor. Permutation importance instead measures how much a chosen validation score worsens when that predictor’s values are shuffled. The first describes how the fitted trees used a predictor to split; the second describes the disruption caused by breaking its association with outcomes in the evaluation data.
| Quantity | What it summarizes | Key qualification |
|---|---|---|
| GLM coefficient | Conditional change in the linear predictor per unit of a predictor | Depends on units, link, and included terms |
| Lasso selection | Whether a penalized fit assigns a nonzero coefficient | Depends on penalty, scaling, and correlated predictors |
| Tree impurity importance | Weighted impurity reduction attributed to split variables | Can favor predictors with many possible split points |
| Permutation importance | Change in a specified score after shuffling a predictor | Depends on score, evaluation data, and predictor dependence |
Scale Changes Coefficient Magnitudes and Penalties
Suppose a model uses a predictor \(x_j\), then the same predictor is recorded in units \(x_j^*=c x_j\), where \(c\ne0\). To preserve the linear predictor, its coefficient must change to \(\beta_j^*=\beta_j/c\), since \(\beta_jx_j=(\beta_j/c)(cx_j)\). Thus the numerical coefficient can change substantially without changing any fitted predictions.
Standardization expresses a predictor in standard-deviation units. If \(z_j=(x_j-\bar{x}_j)/s_j\), where \(s_j\) is the training-sample standard deviation, then its coefficient is \(\gamma_j=\beta_js_j\). In a GLM, \(\gamma_j\) is the change in the linear predictor for a one-standard-deviation increase, conditional on the other terms. This makes a coefficient comparison less sensitive to measurement units, but it does not make the coefficients measures of total predictive importance: correlation, nonlinear relationships, and model specification still matter.
Worked Example: Raw Units Reverse the Apparent Coefficient Scale
A hypothetical logistic model predicts whether a policyholder renews. Its fitted linear predictor contains \(0.04B+0.30A\), where \(B\) is account balance measured in thousands of dollars and \(A\) is age measured in decades. The training standard deviations are 20 balance-units and 0.8 age-units. Compare the raw and standardized coefficient magnitudes, and interpret the one-standard-deviation effects.
Method. First convert each coefficient to its effect per original measurement unit, then multiply the model-scale coefficient by the standard deviation to obtain the effect for one standard deviation.
Per one balance-unit, which is $1,000, the coefficient is \(0.04\). Per one dollar, the coefficient is \(0.04/1000=0.00004\). Per one age-unit, which is ten years, the coefficient is \(0.30\); per one year it is \(0.30/10=0.03\). Comparing per-dollar and per-year coefficients would put \(0.03\) beside \(0.00004\), but that comparison mostly reflects different units.
For balance, the standardized coefficient is \(0.04(20)=0.80\). For age, it is \(0.30(0.8)=0.24\). Thus the fitted linear predictor changes by 0.80 for a one-standard-deviation increase in balance and by 0.24 for a one-standard-deviation increase in age, holding the other model terms fixed.
In logistic regression, the corresponding odds multipliers are \(e^{0.80}\approx2.2255\) and \(e^{0.24}\approx1.2712\). These are model-based conditional effects, not changes in renewal probability of 0.80 or 0.24. As a check, a one-standard-deviation balance increase is 20 balance-units, so \(20(0.04)=0.80\); a one-standard-deviation age increase is 0.8 decades, so \(0.8(0.30)=0.24\). The common mistake is to compare the per-dollar coefficient with the per-year coefficient as if their units matched.
Scaling is particularly consequential for a penalized model. If a predictor is multiplied by \(c\), the coefficient needed to maintain the same fitted values is divided by \(c\), and its absolute-value penalty changes accordingly. Standardizing predictors before penalized fitting makes the penalty more comparable across measurement units; the scaling parameters must be estimated from the training data and then applied unchanged to validation or test data.
Why Correlation Makes Selection and Importance Unstable
When two predictors contain nearly the same information, several models can achieve similar predictions using different allocations of that information. A GLM coefficient is conditional on the other predictors, so correlated predictors can have unstable estimates. A lasso can favor one of the correlated variables because using one may achieve nearly the same fit with less penalty than using both. A tree may split on one correlated variable early, leaving its substitute with few opportunities to receive impurity credit.
The lasso behavior can be seen directly in a simple objective. Suppose two standardized predictors are identical, \(x_1=x_2=z\), the response is \(y=2z\), and the sample is scaled so that \(\frac{1}{n}\sum_i z_i^2=1\). With nonnegative coefficients, the squared-error lasso objective at \(\lambda=0.5\) depends on \(s=\beta_1+\beta_2\):
Differentiating with respect to \(s\) gives \(s-2+0.5=0\), so the minimizing sum is \(s=1.5\). Any nonnegative allocation with \(\beta_1+\beta_2=1.5\) has the same fitted values and penalty. The lasso objective therefore does not distinguish \((1.5,0)\), \((0,1.5)\), and \((0.75,0.75)\) in this exact-duplicate case.
This is a short justification for why selection rankings can disagree even when predictive fits are equivalent: the objective rewards the aggregate fit and penalizes coefficient magnitude, but identical predictors can share the same fitted contribution in multiple ways. With nearly, rather than exactly, correlated predictors, small sample or tuning differences can favor one allocation over another. A selected variable should not automatically be described as the only relevant variable.
Worked Example: Two Equivalent Lasso Allocations
Use the standardized, duplicate-predictor setup above: \(x_1=x_2=z\), \(y=2z\), \(\frac{1}{n}\sum_i z_i^2=1\), and \(\lambda=0.5\). Verify the lasso objective for two different allocations and explain what their selection rankings imply.
Method. Substitute each allocation into the squared-error lasso objective. Since the fitted value is \((\beta_1+\beta_2)z\), both allocations with the same coefficient sum have the same residual error.
For \((\beta_1,\beta_2)=(1.5,0)\), the fitted value is \(1.5z\), so the residual is \(0.5z\). Its mean squared residual is \(0.25\frac{1}{n}\sum_i z_i^2=0.25\). The loss term is therefore \(0.25/2=0.125\), and the penalty is \(0.5(|1.5|+|0|)=0.75\). The total objective is \(0.125+0.75=0.875\).
For \((\beta_1,\beta_2)=(0.75,0.75)\), the fitted value is again \(1.5z\), so the loss is \(0.125\). The penalty is \(0.5(0.75+0.75)=0.75\), giving the same total, \(0.875\). In fact, the objective for a coefficient sum \(s\) is \(\frac12(2-s)^2+0.5s\), whose minimum occurs at \(s=1.5\), confirming that both are minimizing allocations.
The first allocation selects only \(x_1\); the second selects both. A ranking based on “selected versus not selected” therefore depends on the solution among equivalent allocations, not on a difference in fitted predictions. The common mistake is to interpret the first solution as proof that \(x_2\) carries no predictive information, even though it is an exact duplicate of \(x_1\).
Tree Importance Depends on Its Definition
For a tree node \(t\), let \(I(t)\) denote its impurity, \(N_t\) the number of training observations reaching it, and \(t_L,t_R\) its two child nodes. The node’s weighted impurity reduction is
For a fitted tree or forest, impurity importance for variable \(j\) is the sum of \(\Delta_t\) over split nodes using \(j\), often averaged across trees or normalized to sum to one. This formula follows from comparing the parent impurity with the observation-weighted child impurity: a split is credited according to how much it reduces impurity among the full training sample. It does not directly measure what happens when the variable is unavailable at prediction time.
Permutation importance asks a different question. For a loss \(L\), calculate the validation loss before and after shuffling one predictor while leaving other columns fixed. A common score is the average increase:
where \(R\) is the number of shuffles, \(L^{\mathrm{base}}\) is the unpermuted validation loss, and \(L_{j,r}^{\mathrm{perm}}\) is the loss in repeat \(r\). The score depends on the selected loss and evaluation sample. If predictors are correlated, shuffling one breaks their usual relationship. A substitute predictor may still support good predictions, making the shuffled variable appear less important; alternatively, an unrealistic shuffled combination can cause a large loss increase. Neither score alone resolves which correlated predictor is uniquely responsible for predictive signal.
Worked Example: Impurity and Permutation Give Different Rankings
A hypothetical forest uses predictors \(W\), \(R_1\), and \(R_2\). Across its trees, the summed weighted impurity reductions are 50, 30, and 20 units, respectively. On a separate validation set, the baseline mean squared error is 0.20. Repeatedly shuffle each predictor and record the resulting mean squared errors. Calculate both rankings.
Method. Rank impurity importance by summed reduction. Rank permutation importance by the average increase in validation mean squared error over the baseline.
The impurity reductions sum to \(50+30+20=100\). As percentages of the total, \(W\) has \(50/100=50\%\), \(R_1\) has \(30/100=30\%\), and \(R_2\) has \(20/100=20\%\). The impurity ranking is \(W>R_1>R_2\).
Suppose the validation mean squared errors after shuffling \(W\) are 0.26, 0.24, and 0.25. Their increases over 0.20 are 0.06, 0.04, and 0.05, so the mean permutation importance is \((0.06+0.04+0.05)/3=0.05\).
For \(R_1\), suppose the shuffled errors are 0.21, 0.22, and 0.20. The increases are 0.01, 0.02, and 0.00, giving \((0.01+0.02+0.00)/3=0.01\). For \(R_2\), suppose the errors are 0.27, 0.25, and 0.26. The increases are 0.07, 0.05, and 0.06, giving \((0.07+0.05+0.06)/3=0.06\). The permutation ranking is therefore \(R_2>W>R_1\), which differs from the impurity ranking.
The arithmetic check is that each permutation score is an average increase from the same baseline, 0.20; subtracting the baseline first or averaging the shuffled errors and subtracting once gives the same results. The example’s disagreement is not a contradiction: impurity importance credits training split reductions, while permutation importance measures validation loss increases after a particular disruption. The common mistake is to label both numbers simply “importance” and compare their magnitudes as if they were the same quantity.
A Disciplined Interpretation
When rankings disagree, first identify the exact measure, then consider whether the predictors were represented comparably and whether the model assumptions differ. A defensible explanation separates evidence about prediction from evidence about individual variables.
State whether a ranking comes from coefficient magnitude, lasso selection, impurity reduction, permutation loss, or another explicitly defined measure.
Confirm the units and standardization of predictors. For penalized models, verify that scaling was estimated on training data and used consistently at evaluation.
Look for correlated predictors, nonlinear effects, interactions, and differences in the terms or algorithms available to each model.
Report that a measure ranks variables in a particular fitted model and evaluation setup. Do not convert a ranking into a causal claim or a universal ordering.
A useful comparison may report several views side by side rather than forcing them into one list. For example, standardized GLM effects can describe conditional model-scale associations, lasso selection can describe a sparse predictive representation, and permutation scores can describe validation performance disruption. If correlated predictors exchange rank, the analyst can report that predictive signal is shared or difficult to allocate uniquely, rather than asserting that one predictor is definitively unimportant.
Importance rankings are diagnostic summaries, not substitutes for evaluation of the model’s predictive performance or suitability for its intended use. A model may predict well while allocating importance differently among correlated variables. Conversely, a high score under one definition does not guarantee a stable, interpretable, or practically useful effect. These distinctions are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, identify what the importance quantity supports and avoid treating unlike measures as interchangeable.
- A log-link GLM uses a predictor measured in hundreds of units, with coefficient 0.06 and training standard deviation 5 such units. Find the linear-predictor change for a one-standard-deviation increase and the corresponding multiplicative change in the mean. (Answer: 0.30 on the linear-predictor scale; mean multiplier \(e^{0.30}\approx1.3499\).)
- Two exactly identical standardized predictors are fitted by a lasso. One solution has coefficients 1.2 and 0; another has coefficients 0.6 and 0.6. What is the shared fitted contribution, and does the first solution establish that the second predictor has no information? (Answer: both contribute \(1.2z\); no, the allocation is not uniquely attributable between identical predictors.)
- A forest’s summed impurity reductions are 24, 16, and 10 for predictors \(A,B,C\). Calculate their normalized percentages and rank them. (Answer: total \(50\); \(A=48\%\), \(B=32\%\), \(C=20\%\); ranking \(A>B>C\).)
- A validation loss is 0.12. After four shuffles of predictor \(Q\), the losses are 0.15, 0.14, 0.13, and 0.18. Calculate its mean permutation importance. (Answer: increases \(0.03,0.02,0.01,0.06\); mean increase \(0.12/4=0.03\).)
- A continuous predictor has many possible split points and ranks highest in impurity importance, while a correlated predictor ranks highest by permutation importance. Give two reasons not to declare the rankings contradictory. (Answer: the measures summarize different properties—training split reductions versus validation loss disruption—and correlation can redistribute split credit or make shuffling disrupt predictor relationships.)