Variable Importance In Boosted Trees

Why Variable Importance Needs a Definition

A boosted-tree model may use a predictor in many splits, or use it only a few times where it produces substantial reductions in the fitting objective. These are different notions of “importance.” A split-based summary describes how often a predictor was selected; a gain-based summary describes how much objective improvement is attributed to its splits under a stated convention. Neither summary, by itself, says whether a predictor raises or lowers predictions, nor whether changing the predictor would cause an outcome to change.

For an ensemble, each tree is added after the current model has already been updated by earlier trees. A split’s gain is therefore assessed in the context of the model and fitting procedure at that stage. The resulting importance is a description of the fitted ensemble, not a general property of the predictor. The definitions below are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

The word “importance” is not enough to identify a calculation. A table may report raw split counts, percentages of splits, sums of gains, average gain per split, or normalized gain shares. These measures answer different questions and need not produce the same ranking. Always identify which quantity is being reported and which trees, iterations, objective, and gain convention were used.

Split-Based and Gain-Based Summaries

Suppose the model contains \(M\) fitted trees. Let \(\mathcal{S}_j\) be the set of their internal nodes that split on predictor \(j\), and let \(\Delta_s\) denote the gain assigned to node \(s\). If \(K\) is the total number of internal split nodes across the selected trees, then the principal summaries are:

The split frequency \(F_j\) is often displayed as \(100F_j\%\). A gain share can similarly be defined as \(G_j/\sum_k G_k\), provided the total gain in the denominator is positive and all gains use the same convention. Some reports instead show average gain, \(G_j/N_j\), for predictors with \(N_j>0\). The average describes the mean gain at a predictor’s selected splits; it is not the predictor’s total contribution across the ensemble.

Gain is implementation-dependent. For a regression tree using squared-error loss, it may be defined as the reduction in residual sum of squares. In gradient-boosting software, it may instead use an objective-specific split score, potentially including weights, gradients, Hessians, or regularization terms. Thus, a reported gain is meaningful only with its fitting objective and software convention understood. For the hand calculations below, the stated gain definition is the unpenalized reduction in the specified loss.

Why Squared-Error Gain Has This Form

For a regression tree using squared-error loss, suppose a parent node contains \(n=n_L+n_R\) observations with response mean \(\bar{y}\). A proposed split creates left and right child means \(\bar{y}_L\) and \(\bar{y}_R\). The parent predicts \(\bar{y}\) for every observation in that node, while each child predicts its own mean. Define each node’s loss as the sum of squared deviations from that node’s mean. Under these assumptions, the split gain is the parent loss minus the sum of the child losses.

For the first equality, write each deviation as \(y_i-\bar{y}=(y_i-\bar{y}_L)+(\bar{y}_L-\bar{y})\) in the left child, and analogously in the right child. On squaring and summing within a child, the cross term is zero because deviations from that child’s mean sum to zero. For the second equality, the parent mean is \(\bar{y}=(n_L\bar{y}_L+n_R\bar{y}_R)/n\). Hence \(\bar{y}_L-\bar{y}=(n_R/n)(\bar{y}_L-\bar{y}_R)\) and \(\bar{y}_R-\bar{y}=-(n_L/n)(\bar{y}_L-\bar{y}_R)\). Substitution gives the final term. Therefore the squared-error gain is nonnegative and equals that final term. This identity assumes unweighted squared-error loss and child predictions equal to their sample means; it is not a general gain formula for every boosting objective.

What Importance Values Do and Do Not Establish

Split frequency and gain are useful for describing how a particular fitted model used its available predictors. A high split count means a predictor was selected at many internal nodes, not necessarily that those nodes produced large improvements. A high total gain means its selected splits accumulated a large amount under the stated gain convention; it does not reveal the direction of prediction changes. For logistic boosting, for example, a gain based on the fitting objective is not automatically a change of the same size in predicted probability.

The values are conditional on modeling choices. Changing the training sample, tree depth, number of boosting iterations, learning rate, regularization, or available predictors can change which splits are selected and their gains. Correlated predictors may offer similar information; a tree can select one in a particular fit and assign it gain that another fit assigns to its correlate. A predictor with many possible split points or categories may also have more opportunities to be selected, depending on the implementation and its handling of predictors.

Most importantly, neither split count nor gain supports a causal conclusion. The fact that a predictor appears frequently or receives substantial gain does not show that intervening on it would change the outcome. The model may be using a proxy, exploiting an association created by confounding or selection, or capturing a predictive relationship that is not causal. A causal claim requires an appropriate study design and assumptions beyond a predictive importance table.

Reading an Importance Table Carefully

When reporting an importance summary, state the measure and its scope. For example, “split frequency among the first 120 trees” is more precise than “importance.” If reporting gain, clarify whether it is a raw sum, an average per split, or a normalized share, and whether the implementation’s gain includes objective-specific or regularization terms. Software packages do not necessarily use identical definitions: an implementation may report relative influence or normalize a gain-based total, while another may expose split counts and objective-score gains.

A disciplined interpretation keeps three questions separate:

• Usage: How often did the fitted trees split on this predictor?
• Objective reduction: How much gain was assigned to those splits under the stated definition?
• Substantive meaning: What, if anything, can be concluded about direction, mechanism, or causation? Importance alone does not answer these questions.

Importance summaries are most useful as compact diagnostics or communication aids when their definitions and limitations are explicit. They can help identify predictors the fitted ensemble relied on, but should not be presented as a universal ranking of variables or as proof of causal influence.