Why Importance Stability Matters
A forest can produce a ranked list of predictors, but a single ranking does not show whether that ordering would persist under a slightly different sample or forest fit. A variable ranked first in one fit may rank lower after resampling, while several predictors may remain near the top across many fits. Assessing this variation helps distinguish a persistent predictive signal from an ordering that is sensitive to the data or modeling choices.
Stability is not a new importance measure. It is an assessment of how consistently a chosen importance measure identifies or orders predictors under a defined set of repeated analyses. The interpretation depends on what is varied: resampling the observations assesses sensitivity to the sampled data; refitting with different random seeds assesses algorithmic variability; changing tuning settings assesses sensitivity to modeling choices. These questions should be kept distinct.
The earlier material on permutation importance and conditional permutation importance provides possible importance measures. Choose one measure for a comparison and use it consistently. For example, a permutation importance for a regression forest may be defined as the increase in validation mean squared error after permuting a predictor. If the same importance calculation is used for every fit, differences between results can be interpreted as variation across the chosen resamples or fits. If the measure itself changes, the comparison combines two sources of variation.
Three Useful Summaries
Let \(p\) be the number of predictors, and let \(r_{bj}\) denote the rank of predictor \(j\) in repeated analysis \(b\), with rank 1 assigned to the largest importance. A stability analysis can summarize ranks, top-\(k\) membership, or agreement between complete rankings.
A mean rank near 1 indicates consistently high placement on average, but it can hide occasional large movements. Top-\(k\) frequency answers a different question: how often does a predictor appear in a shortlist of a chosen size? Neither summary alone describes every feature of the rank distribution, so reporting both, or also reporting a range or spread of ranks, can be useful.
To compare two complete rankings, Spearman’s rank correlation measures the association between their ranks. It ranges from -1 to 1: values near 1 indicate similar ordering, values near 0 indicate little rank association, and values near -1 indicate largely reversed ordering. The shortcut formula below applies when there are no tied ranks.
When ties occur, use the Pearson correlation of the rank values, assigning tied predictors their average rank, rather than applying the no-ties shortcut without adjustment. The resulting value still summarizes rank association, but the shortcut identity does not hold as stated.
Why the Rank-Correlation Formula Works
For two rankings without ties, each rank list is a permutation of \(1,\ldots,p\). Both lists therefore have the same mean rank, \((p+1)/2\), and the sum of squared deviations from that mean is \(p(p^2-1)/12\). Write the two ranks for predictor \(j\) as \(r_{1j}\) and \(r_{2j}\). The squared rank differences satisfy
Because both rank lists have the same sum of squares, their Pearson correlation is their centered cross-product divided by \(p(p^2-1)/12\). Substituting the displayed identity into that correlation gives
Thus the shortcut is a direct consequence of correlation applied to ranks. It measures ordering agreement, not agreement in the absolute sizes of importance values.
Worked Example: Finding Persistent Top Predictors Across Resamples
An analyst fits the same type of hypothetical claim-frequency forest on three training resamples. Using the same importance method and evaluation metric, the resulting ranks for four predictors are as follows. Rank 1 is most important. Summarize the top-two frequency and mean rank for each predictor.
| Predictor | Resample 1 | Resample 2 | Resample 3 |
|---|---|---|---|
| Territory | 1 | 1 | 3 |
| Vehicle age | 2 | 3 | 1 |
| Annual mileage | 3 | 2 | 2 |
| Payment plan | 4 | 4 | 4 |
Method. For each predictor, count its appearances in ranks 1 and 2, then average its three ranks. The top-two frequency uses \(B=3\) and \(k=2\).
Territory: it is in the top two twice, so \(f_{\text{Territory},2}=2/3=0.667\). Its mean rank is \((1+1+3)/3=5/3=1.667\).
Vehicle age: it is in the top two in resamples 1 and 3, so its top-two frequency is \(2/3=0.667\). Its mean rank is \((2+3+1)/3=6/3=2.000\).
Annual mileage: it is in the top two in resamples 2 and 3, so its top-two frequency is also \(2/3=0.667\). Its mean rank is \((3+2+2)/3=7/3=2.333\).
Payment plan: it is never in the top two, so its top-two frequency is \(0/3=0\). Its mean rank is \((4+4+4)/3=4.000\).
Territory has the best mean rank, but all three of the first predictors appear in the top two in two of the three resamples. The summary supports saying that these predictors are frequently highly ranked, while their exact ordering is not fully stable. Common mistake avoided: declaring a predictor uniquely dominant from one fit, despite changes in its rank across resamples.
Separating Sampling and Fitting Variation
A useful design starts by naming the source of variation being studied. To assess sensitivity to sampled observations, refit the forest on multiple resamples of the development data. To assess stochastic fitting variation, hold the data and tuning settings fixed and repeat the fit with different random seeds. To assess tuning sensitivity, compare prespecified settings while keeping the data partitions, importance definition, and evaluation metric comparable.
These designs answer different questions. If both the data and random seed change at once, the results describe their combined variability; they do not identify which source caused a ranking change. A paired comparison—changing one factor while holding the others fixed—is easier to interpret. For tuning comparisons, the candidate settings should be evaluated under the same resampling scheme, so that a difference is not simply due to one setting receiving easier validation samples.
Importance values must also be on a comparable scale. For a loss-increase measure, use the same loss definition, response scale, evaluation observations, and permutation procedure for each fit. If the evaluation samples differ, changes in importance can reflect both the forest and the difficulty or composition of the evaluation data. Repeated use of a final holdout set to select predictors or settings can also make that set part of model development; reserve an untouched evaluation set for a final assessment when one is available.
Worked Example: Comparing Two Rankings With Spearman Correlation
Two repeated forest fits rank four predictors as follows: fit A ranks them \(W:1,\ X:2,\ Y:3,\ Z:4\), while fit B ranks them \(W:2,\ X:1,\ Y:4,\ Z:3\). Calculate Spearman’s rank correlation and interpret it.
Method. There are \(p=4\) predictors and no ties, so use the no-ties shortcut. The rank differences \(d_j=r_{Aj}-r_{Bj}\), in the order \(W,X,Y,Z\), are \(-1,1,-1,1\). Their squares are \(1,1,1,1\), giving \(\sum_j d_j^2=4\).
Substitute into the formula:
The rankings have a positive but imperfect association: the ordering is not reversed, but positions change enough that the fits do not agree closely. This value does not say that the importance magnitudes are 60% alike. Common mistake avoided: interpreting rank correlation as agreement in the raw importance values.
Assessing Sensitivity to Tuning Choices
Importance stability across tuning settings is often more informative than stability across random seeds alone. A predictor that remains highly ranked over reasonable settings is less sensitive to those particular modeling choices. However, this is not proof that it is universally important: the conclusion is limited to the tested settings, data, and importance definition.
Suppose a regression forest’s permutation importance is measured as the increase in validation mean squared error after permutation. This interpretation is consistent only if the same response units, validation observations, and calculation are used for each setting. The following example compares three settings using that common scale.
Worked Example: Identifying Tuning-Sensitive Rankings
Three candidate values of \(mtry\) are evaluated on the same validation data using the same regression-forest importance calculation. The entries are increases in validation mean squared error; larger values indicate greater measured importance for that fit. Determine each predictor’s rank in every setting, mean rank, and top-two frequency.
| Predictor | Setting A | Setting B | Setting C |
|---|---|---|---|
| Repair history | 0.08 | 0.09 | 0.05 |
| Equipment age | 0.07 | 0.04 | 0.02 |
| Operating hours | 0.03 | 0.05 | 0.06 |
Setting A ranks: \(0.08>0.07>0.03\), so repair history is rank 1, equipment age rank 2, and operating hours rank 3.
Setting B ranks: \(0.09>0.05>0.04\), so repair history is rank 1, operating hours rank 2, and equipment age rank 3.
Setting C ranks: \(0.06>0.05>0.02\), so operating hours is rank 1, repair history rank 2, and equipment age rank 3.
Repair history has mean rank \((1+1+2)/3=4/3=1.333\), and top-two frequency \(3/3=1.000\). Equipment age has mean rank \((2+3+3)/3=8/3=2.667\), and top-two frequency \(1/3=0.333\). Operating hours has mean rank \((3+2+1)/3=6/3=2.000\), and top-two frequency \(2/3=0.667\).
Repair history is consistently in the top two, while operating hours moves from third to first. The rank summaries reveal this movement even though its mean rank is second. Common mistake avoided: treating a change in rank as evidence of a meaningful change in the absolute importance scale; here the ranking is the direct comparison, and the importance values are comparable only because the evaluation setup was held constant.
Interpreting Stability Carefully
A practical report should state how many analyses were run, what changed between them, which importance measure was used, and how rankings were summarized. For example, “Across 30 resamples, predictor A appeared in the top three in 26 fits” is more informative than saying “A was important.” If rank correlations are reported, summarize their distribution rather than relying on a single pair of fits; a median and range can communicate typical agreement and its variation.
Stability does not establish causality, and an unstable ranking does not necessarily mean the predictors have no predictive value. Predictors that carry overlapping information can trade places across fits, and small differences in importance can produce rank changes. Conversely, a consistently high rank is evidence of reproducibility under the conditions tested, not a guarantee of performance in every population or future sample.
Treat ties deliberately. If importance values are exactly equal, state the tie rule; average ranks are one standard choice. If values are very close, even a no-ties ranking can imply more separation than the values justify. Report the values or a suitable measure of their spread alongside ranks when that distinction matters. Stability conclusions should be proportional to the amount of resampling and the range of modeling choices actually examined.
Specify whether the analysis concerns resampling, random fitting variation, tuning sensitivity, or a deliberately combined source of variation.
Use a consistent importance definition and metric. For paired comparisons, keep data partitions and all non-target fitting choices fixed.
Report mean ranks, top-\(k\) frequencies, and, where useful, rank correlations or rank ranges.
Describe which predictors are repeatedly highly ranked and which move substantially, specifying that the result is conditional on the analyses performed.
Check Your Understanding
For each problem, calculate the requested stability summary and interpret it in context.
- In four resamples, a predictor has ranks 1, 3, 2, and 4 among six predictors. Find its mean rank and its top-three frequency. (Answer: mean rank \(=2.5\); top-three frequency \(=3/4=0.750\).)
- Two rankings of five predictors have squared rank differences summing to 10, with no ties. Calculate Spearman’s rank correlation. (Answer: \(\rho_s=1-\frac{6(10)}{5(25-1)}=0.500\).)
- A predictor appears in the top three in 18 of 24 repeated fits. Calculate its top-three frequency and state what that result does and does not establish. (Answer: \(18/24=0.750\); it was highly ranked in 75.0% of these fits, which establishes consistency only under the tested data and fitting conditions.)
- Across four settings, predictor M has ranks 1, 2, 1, 4, while predictor N has ranks 2, 1, 2, 1. Calculate both mean ranks and top-two frequencies. Which appears more consistently near the top? (Answer: M has mean rank \(2.0\) and top-two frequency \(3/4=0.750\); N has mean rank \(1.5\) and top-two frequency \(4/4=1.000\), so N is more consistently in the top two.)
- Two analysts compare importance values from forests evaluated on different response scales and then report a large rank correlation. What must they check before interpreting that correlation as evidence of stability? (Answer: They must establish that the importance definition and evaluation scale are comparable; if the scales or metrics differ, the rank comparison does not isolate stability across fits.)