What Feature Importance Measures
A boosted tree ensemble makes predictions by adding the contributions of many trees. Each split in a tree uses a predictor to divide observations into groups. Split-based feature importance summarizes how the fitted ensemble used those predictors: for example, how often a feature appeared in a split, or how much the splits using it improved the tree-fitting criterion.
These summaries describe a fitted model, not an intrinsic property of a predictor. A feature can rank highly because it repeatedly helps the algorithm reduce its chosen training objective. That does not, by itself, establish that the feature is important in every reasonable model, that changing it would change the outcome, or that it has a causal effect.
For gain-based importance, let \(G_{tj}\) be the gain at split \(j\) in tree \(t\), and let \(x_{tj}\) be the feature used at that split. The total gain for feature \(k\) is
Here, \(\mathbf{1}(x_{tj}=k)\) equals 1 when split \(j\) in tree \(t\) uses feature \(k\), and 0 otherwise. A normalized gain importance divides each feature's total by the sum across all features. If that sum is positive, the normalized scores sum to 1, or 100% when reported as percentages. Normalization changes the scale, not the underlying ranking.
Gain for a Squared-Error Split
Consider a regression tree fitted using squared-error loss, with equally weighted observations in a node. The prediction in each node is its sample mean. For a proposed split, the unregularized gain is the parent's sum of squared errors minus the two children's combined sum of squared errors. A split with a larger gain reduces the fitting criterion more, subject to any additional constraints or penalties used by the algorithm.
For a node with outcomes \(y_1,\ldots,y_n\), its squared-error loss at the mean \(\bar y\) is \(\sum_{i=1}^n(y_i-\bar y)^2\). Compute this for the parent and both children.
The split gain is the parent loss minus the left-child loss minus the right-child loss. A positive gain means the split reduces squared error.
For total-gain importance, add this gain to the score of the feature used for that split. Repeat over the ensemble.
The familiar shortcut for this gain follows from the within-group sum-of-squares identity. Let the parent have \(n\) observations and mean \(\bar y\); let the children have sizes \(n_L,n_R\) and means \(\bar y_L,\bar y_R\), with \(n_L+n_R=n\). The parent mean satisfies \(n\bar y=n_L\bar y_L+n_R\bar y_R\). For each child, expand an observation's deviation from the parent mean around that child's mean. The cross-product terms sum to zero because deviations from a child mean sum to zero. Therefore,
The last two terms are exactly the reduction in squared error. Substituting the expression for \(\bar y\) and simplifying gives
This formula applies to an unweighted squared-error split with child means as predictions. It is not a general formula for every boosting objective. In boosting, trees may be fitted to gradients or other quantities, and implementations may calculate gain from an objective-specific criterion, apply regularization, or use observation weights. Use the gain definition associated with the stated fitting method.
Worked Example: Calculating Total and Normalized Gain
A hypothetical squared-error boosting model contains three trees. The recorded split gains are: Tree 1 uses feature \(A\) with gains 8 and 4, and feature \(B\) with gain 3; Tree 2 uses \(B\) with gain 6, \(A\) with gain 2, and \(C\) with gain 5; Tree 3 uses \(C\) with gain 4 and \(B\) with gain 1. Calculate each feature's total-gain importance and its normalized percentage.
Method. Sum the gains across all splits using each feature, then divide by the total gain across features.
For \(A\), the gains are \(8,4,2\), so \(I_A^{\text{gain}}=8+4+2=14\). For \(B\), they are \(3,6,1\), so \(I_B^{\text{gain}}=3+6+1=10\). For \(C\), they are \(5,4\), so \(I_C^{\text{gain}}=5+4=9\). The ensemble's total gain is \(14+10+9=33\).
The normalized scores are \(14/33=0.4242\), \(10/33=0.3030\), and \(9/33=0.2727\), to four decimal places. As percentages, these are 42.42%, 30.30%, and 27.27%, respectively. Their displayed percentages total 99.99% because of rounding; before rounding they total 100%.
Check. The unnormalized gains add to the reported total: \(14+10+9=33\). Common mistake avoided: these percentages are shares of total split gain, not shares of the model's predictive accuracy or shares of explained outcome variation.
Frequency, Average Gain, and Cover
The same ensemble can receive quite different importance rankings depending on the summary chosen. Split count gives each split equal weight, regardless of how much it improves the fitting criterion. Average gain gives the mean gain among a feature's splits, so a feature used once with a large gain can rank above one used many times with modest gains. Cover summarizes the data associated with splits; whether it means a count, a weighted count, or a particular node-level quantity depends on the implementation.
| Measure | What it summarizes | What it does not say by itself |
|---|---|---|
| Split count | How many splits use the feature | Whether those splits produced large improvements |
| Total gain | The sum of the gains at the feature's splits | How much performance would fall if the feature were removed and the model refitted |
| Average gain | The mean gain per split using the feature | How often the feature was useful across all observations or models |
| Cover | A software-defined measure of data associated with its splits | That the feature has a large effect on each observation |
Worked Example: Why Split Count and Total Gain Can Disagree
A hypothetical ensemble has six splits. Feature \(M\) is used four times, with gains \(1,1,1,1\). Feature \(N\) is used twice, with gains \(7,5\). Find split-count, total-gain, and average-gain comparisons.
Method. Count the splits for frequency, sum gains for total gain, and divide each total by that feature's own split count for average gain.
Feature \(M\) has split count 4, total gain \(1+1+1+1=4\), and average gain \(4/4=1\). Feature \(N\) has split count 2, total gain \(7+5=12\), and average gain \(12/2=6\). Across both features, total gain is \(4+12=16\); normalized total-gain importance is \(4/16=25\%\) for \(M\) and \(12/16=75\%\) for \(N\).
Thus \(M\) appears more often, but \(N\) contributes more total gain and has a larger average gain per split. Check. The counts add to six and the gains add to 16. Common mistake avoided: a higher split count does not imply a higher gain-based importance.
Interpreting Importance with Correlated Predictors
When two predictors carry similar information, a tree can often use either one to make a useful split. Which predictor is selected can depend on small differences in the training sample, candidate split points, variable availability, or fitting choices. Once one correlated predictor has been used, the other may provide less additional improvement in later splits. As a result, importance may be concentrated on one feature, divided between them, or reassigned across fitted models even when predictive performance is similar.
A low score for one member of a correlated pair therefore does not establish that the feature contains no useful information. It may be redundant with the other predictor in that fitted model. Conversely, a large score does not show that the feature is uniquely necessary: another correlated feature might substitute for it. The ranking should be read alongside the predictor definitions, the data-generating context, and the model's performance on data not used for fitting.
Worked Example: Importance Shared by Correlated Risk Features
A hypothetical insurer fits a boosted model for claim severity using building age and roof age. In one fitted ensemble, building age has two splits with gains 12 and 8; roof age has one split with gain 4. These features are strongly correlated in the training data. Calculate the gain shares and interpret them.
Method. Sum each feature's gains and divide by the combined gain. Building age has total gain \(12+8=20\); roof age has total gain 4. Combined gain is \(20+4=24\).
The normalized gain shares are \(20/24=0.8333\) and \(4/24=0.1667\), or 83.33% and 16.67% to two decimal places. The fitted ensemble assigned most of its recorded split gain to building age. This is not evidence that building age is inherently more important than roof age: a different fitted model could use roof age for a similar partition when the two variables convey overlapping information.
Check. The shares add to 100.00% after rounding, and the raw gains sum to 24. Common mistake avoided: interpreting the 83.33% share as a statement that building age causes 83.33% of claim severity.
Importance Is Not a Causal Effect
A feature can be useful for prediction without causing the outcome. Split-based importance describes how the fitted algorithm used a variable to improve its objective, given the available predictors and data. Causal interpretation requires a different question: what would happen to the outcome under a well-defined intervention that changed the feature, compared with an appropriate alternative? A split score does not answer that question.
For example, a hypothetical insurer may arrange pre-loss inspections more often for properties already suspected to have higher risk. An inspection flag could help a boosted model predict later claim costs because it reflects the insurer's selection process or information about underlying risk. Its importance would not show that arranging an inspection increases claim costs. The direction of the relationship, the timing of the measurements, confounding factors, and a defensible causal design all matter.
Worked Example: A Predictive Split Does Not Estimate an Intervention Effect
In a hypothetical data set of eight policies, the target is claim cost in thousands of dollars. Four policies with no pre-loss risk flag have costs \(0,0,2,2\); four with the flag have costs \(8,8,10,10\). A squared-error tree considers a split on the flag. Calculate the split gain, then state what it does and does not establish.
Method. Calculate the parent and child sums of squared deviations from their respective means, then subtract child loss from parent loss.
The parent mean is \((0+0+2+2+8+8+10+10)/8=40/8=5\). Its squared-error loss is \(25+25+9+9+9+9+25+25=136\).
The no-flag group's mean is \((0+0+2+2)/4=1\), and its loss is \(1+1+1+1=4\). The flag group's mean is \((8+8+10+10)/4=9\), and its loss is also \(1+1+1+1=4\). The split gain is \(136-4-4=128\) in squared thousands-of-dollars units.
The flag provides a large predictive reduction in squared error for this split. It does not establish that intervening to set the flag would change claim costs. In this scenario the flag may identify properties already selected as higher risk; the observed association can reflect selection or underlying risk. Check. The split's child losses total 8, so \(136-8=128\). Common mistake avoided: treating a high split gain as a causal effect or as the predicted change under an intervention.
Common Interpretation Traps
- Confusing frequency with contribution. A feature used often can have small gains at each split. State whether the reported score is count, total gain, or another measure.
- Comparing scores without checking their scale. Raw gain depends on the objective and its scale. Normalized scores ease within-model comparison, but do not make scores automatically comparable across different models or data sets.
- Treating correlated-feature rankings as stable. Correlated predictors can substitute for one another, so a single ranking may conceal shared predictive information.
- Calling importance causal. A fitted split score is a predictive summary, not an estimate of the effect of changing a predictor.
- Assuming an importance score measures unique value. Total gain is accumulated over splits in the fitted ensemble; it is not the performance change from removing a feature and refitting.
Check Your Understanding
For each problem, identify the requested importance measure and check the arithmetic before interpreting it.
- A fitted model has gains 3 and 5 for feature \(R\), and gains 4 and 4 for feature \(S\). Find total gain and normalized total-gain percentage for each. (Answer: \(R=8\), \(S=8\); 50% each.)
- Feature \(U\) appears in five splits with total gain 15. Feature \(V\) appears in three splits with total gain 18. Which has more splits, and which has greater average gain per split? (Answer: \(U\) has more splits; average gains are 3 for \(U\) and 6 for \(V\), so \(V\) has the greater average.)
- An unweighted squared-error split has 6 observations in the left child with mean 2 and 4 in the right child with mean 7. Use the split-gain shortcut to calculate the gain. (Answer: \((6\cdot4/10)(7-2)^2=60\).)
- Two highly correlated predictors have normalized gain shares of 70% and 30% in one model. A second model has similar validation performance but reverses those shares. What is the most defensible interpretation? (Answer: the fitted models allocate split gain differently between correlated predictors; the shares do not establish that one predictor is uniquely necessary or causal.)
- A feature describing whether a property received a pre-loss inspection has high total gain in a claim-cost model. Inspections are more commonly arranged for properties already believed to be risky. Does the gain estimate the effect of arranging an inspection? (Answer: no; it indicates predictive use in that fitted model, not a causal intervention effect.)
These distinctions are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus: calculate a model summary accurately, then avoid claiming more than that summary supports.