Why Check Importance Stability?
A random forest can produce a ranking of predictors, but a ranking from one fitted forest does not show whether that ordering would persist under another fit. Bootstrap samples, feature subsampling, and other random choices can change which predictors are used and how much importance is assigned to them. In turn, the same predictor may move several places in the ranking even when the fitting procedure has not changed.
Stability analysis compares importance results across repeated fits made under a clearly specified design. It helps distinguish a robust ordering from one that depends heavily on the particular random fit. It does not, by itself, establish that a predictor is causal, statistically significant, or useful for every prediction task. A rank is relative: a predictor can move upward because another predictor moved downward, even if its own importance barely changed.
For example, if two predictors have importance values of 10 and 9 in one fit and 2 and 1 in another, their ranks are identical across fits but their importance magnitudes are not. Conversely, two predictors whose importance values remain close to 5 might swap ranks repeatedly because their values are nearly tied.
Designing a Fair Comparison
Before comparing ranks, specify what is being repeated. To assess algorithmic variability, hold the observed data and modeling choices fixed and refit the forest with different random seeds. This captures variability from the forest-fitting randomness, such as bootstrap samples and feature selection at splits. It does not capture uncertainty from having observed a different sample of cases.
To assess sampling variability, fit forests on multiple resampled versions of the data, using the same modeling procedure for each. This changes the cases as well as the forest randomness. When interpreting the result, keep these two sources of variation distinct. If the question concerns both, use a design that repeats the data resampling and the forest fitting, and state that both sources are represented.
For a fair comparison, keep the response, predictor definitions, importance method, loss or scoring metric, and major forest settings consistent. In particular, importance values based on different metrics or scales should not be compared as if they were directly interchangeable. If using out-of-bag (OOB) permutation importance, apply the same importance definition and error metric in each fit. A rank comparison is less sensitive to changes in scale than a comparison of raw values, but it cannot correct for a change in what the importance measure means.
Decide whether the goal is to assess forest randomization on fixed data, sampling variability across resampled data, or both.
Use the same response, predictors, importance definition, scoring metric, and specified forest settings in every fit.
For every fit, retain each predictor’s importance and its rank; keep the underlying values because ranks alone discard magnitude information.
Report useful rank summaries and examine them alongside importance magnitudes, predictor relationships, and the repeat scheme.
Useful Summaries of Rank Stability
Three summaries answer different practical questions. The mean rank describes the predictor’s average position. Its range or interquartile range describes how much its position varies. The top-\(k\) frequency is the proportion of fits in which it appears among the first \(k\) predictors. For predictor \(j\), the top-\(k\) frequency is the number of fits with \(R_{jb}\leq k\), divided by \(B\).
Here \(\mathbf{1}(\cdot)\) equals 1 when the condition is true and 0 otherwise. A low mean rank and a high top-\(k\) frequency suggest that a predictor often ranks near the top, but they do not show whether the predictor has a large or consistently positive importance. Inspecting the importance values and their signs is essential, especially for permutation importance, which can be zero or negative.
To compare two complete rankings, Spearman’s rank correlation is a descriptive summary of their agreement. When the ranks have no ties, it can be calculated from the squared rank differences. A value near 1 indicates similar orderings, a value near 0 indicates little monotone agreement, and a value near \(-1\) indicates largely reversed orderings. With ties, calculate the ordinary correlation of the rank values, using average ranks for tied predictors; the simplified formula below assumes no ties.
The formula follows from the ordinary correlation of ranks. With no ties, both rank lists contain \(1,\ldots,p\), so each has mean \((p+1)/2\) and the same sum of squared deviations, \(p(p^2-1)/12\). The sum of squared differences is \(\sum_j(R_{j1}-R_{j2})^2\). Expanding this expression and using the equal rank means and equal sums of squares gives the correlation as one minus \(6\sum_jd_j^2/[p(p^2-1)]\). Thus small squared rank differences produce a correlation close to 1; large differences reduce it.
This coefficient is descriptive here, not a test that the rankings are equal. In particular, a high overall correlation can coexist with a consequential swap among the few highest-ranked predictors. Report the ranks or top-\(k\) frequencies as well when decisions depend on the leading predictors.
Worked Example: Comparing Top-Rank Persistence Across Five Fits
A hypothetical insurer fits five regression forests to the same claim-severity data, holding the importance definition and forest settings fixed while changing the random seed. The table gives the resulting predictor ranks. Rank 1 is most important. Calculate mean ranks and top-two frequencies, then interpret the leading predictors.
| Predictor | Fit 1 | Fit 2 | Fit 3 | Fit 4 | Fit 5 |
|---|---|---|---|---|---|
| Repair-cost index | 1 | 1 | 2 | 1 | 3 |
| Vehicle age | 2 | 2 | 1 | 2 | 1 |
| Territory band | 3 | 4 | 3 | 4 | 2 |
| Policy duration | 4 | 3 | 4 | 3 | 4 |
Method. Add each row’s five ranks and divide by 5. For the top-two frequency, count ranks of 1 or 2 and divide by 5.
Repair-cost index. Its rank total is \(1+1+2+1+3=8\), so its mean rank is \(8/5=1.6\). It is in the top two in Fits 1 through 4, giving \(4/5=0.8\).
Vehicle age. Its total is \(2+2+1+2+1=8\), so its mean rank is \(8/5=1.6\). It is in the top two in all five fits, giving \(5/5=1.0\).
Territory band. Its total is \(3+4+3+4+2=16\), so its mean rank is \(16/5=3.2\). It appears in the top two once, giving \(1/5=0.2\).
Policy duration. Its total is \(4+3+4+3+4=18\), so its mean rank is \(18/5=3.6\). It never appears in the top two, giving \(0/5=0\).
Vehicle age has the most persistent top-two position, even though its mean rank ties with the repair-cost index. That tie in mean rank hides a difference in top-two frequency. This example gives ranks only: without the underlying importance values, it does not establish whether either leading predictor has a large or reliably positive importance. The common mistake is to treat a mean-rank tie as proof of identical stability; the top-two counts show otherwise.
Worked Example: Measuring Agreement Between Two Rankings
A second hypothetical analysis compares two forest fits ranking five predictors of equipment downtime. Fit A ranks them, in predictor order, as \(1,2,3,4,5\). Fit B assigns ranks \(2,1,4,3,5\). Calculate Spearman’s rank correlation and state what it indicates.
Method. There are no ties, so use the rank-difference formula. In predictor order, the differences are \(d_j=R_{jA}-R_{jB}\): \(1-2=-1\), \(2-1=1\), \(3-4=-1\), \(4-3=1\), and \(5-5=0\).
Square the differences: \(1,1,1,1,0\). Their sum is \(4\). With \(p=5\), \(p(p^2-1)=5(25-1)=120\). Therefore \(\rho_s=1-6(4)/120=1-24/120=0.8\).
The two rankings have substantial overall agreement, with two adjacent pairs switching order. This does not mean the leading predictors are necessarily interchangeable: the first two predictors have indeed swapped. The common mistake is to call \(0.8\) perfect agreement or to infer that every position is stable. The coefficient summarizes all five rank differences and must be read alongside the rankings themselves.
Interpreting Rank Changes
A change in rank is a signal to investigate, not a diagnosis by itself. Several explanations can produce the same observed change:
- Forest randomization or sampling variation. A predictor near a competitor in importance can switch places across fits, even when the overall ordering is broadly similar.
- Correlated predictors. Predictors that carry overlapping information can share or exchange importance. A forest may rely more on one proxy in one fit and more on another in a different fit. Individual rankings may therefore be unstable even when the shared predictive information is consistently useful.
- Weak predictive signal. Predictors with importance values close to zero may have ranks that move substantially because small changes in the fitted forest can reorder them. A rank near the top within a set of uniformly weak predictors is not evidence of strong predictive value.
- Different importance scales or procedures. Changing the metric, data used for evaluation, or importance definition can change the result. Such changes should not be mistaken for instability under a fixed procedure.
For correlated predictors, report the individual rankings but avoid interpreting every swap as evidence that one variable has displaced the other in a substantive sense. A separate group-level assessment may help answer whether the correlated set carries predictive information. The group procedure must be defined explicitly; adding individual importance values is not generally equivalent to measuring the importance of the group jointly.
Worked Example: Correlated Predictors and Near-Zero Importance
A hypothetical forest study records OOB permutation importance as the increase in mean squared error after permuting one predictor. Positive values indicate increased error after permutation; values near zero indicate little measured contribution under this procedure, and negative values mean the permuted version happened to score better. Importance is reported in the same squared-response units for all four fits. Two predictors, \(A\) and \(B\), are strongly correlated; \(C\) and \(D\) are candidate weak predictors. The table gives importance values.
| Predictor | Fit 1 | Fit 2 | Fit 3 | Fit 4 |
|---|---|---|---|---|
| A | 8 | 2 | 7 | 1 |
| B | 2 | 8 | 1 | 7 |
| C | 0.2 | -0.1 | 0.1 | -0.2 |
| D | 0.1 | 0.2 | -0.2 | 0.3 |
Step 1: Rank within each fit. In Fit 1, \(8>2>0.2>0.1\), so the ranks for \(A,B,C,D\) are \(1,2,3,4\). In Fit 2, \(8>2>0.2>-0.1\), so the ranks are \(2,1,4,3\). In Fit 3, \(7>1>0.1>-0.2\), giving \(1,2,3,4\). In Fit 4, \(7>1>0.3>-0.2\), giving \(2,1,4,3\).
Step 2: Summarize ranks. For \(A\), the ranks total \(1+2+1+2=6\), so its mean rank is \(6/4=1.5\). For \(B\), the total is \(2+1+2+1=6\), also giving \(1.5\). Each is top-ranked in two of four fits, so each has a top-one frequency of \(2/4=0.5\). For \(C\), the total is \(3+4+3+4=14\), giving mean rank \(14/4=3.5\). For \(D\), the total is \(4+3+4+3=14\), also giving \(3.5\).
Step 3: Check importance magnitudes. The mean importance of \(A\) is \((8+2+7+1)/4=18/4=4.5\), and \(B\)'s is also \((2+8+1+7)/4=18/4=4.5\). Their individual ranks alternate, while their combined importance is \(10\) in Fits 1 and 2 and \(8\) in Fits 3 and 4. By contrast, \(C\)'s mean importance is \((0.2-0.1+0.1-0.2)/4=0\), and \(D\)'s is \((0.1+0.2-0.2+0.3)/4=0.1\).
Interpretation. The repeated exchange between \(A\) and \(B\), alongside their positive values, is consistent with correlated predictors sharing importance; it does not show that either predictor is unimportant. The low, sometimes negative values for \(C\) and \(D\) suggest little stable individual contribution in these fits, despite their changing order. These four fits are descriptive evidence, not a formal guarantee about future samples. A common mistake is to interpret every rank change as a change in predictive value. Here the substantial movement in ranks for \(A\) and \(B\) occurs alongside positive importance values for both.
Common Traps and Practical Conclusions
- Ranking raw values from incomparable fits. If the importance definition or scale changes, apparent movements may not reflect forest instability. Keep the procedure fixed or explain the change.
- Using ranks without magnitudes. Ranks conceal distances. Importance values of 10 and 9 have the same ordering as 10 and 0.01, but those gaps tell different stories.
- Calling a single random-seed comparison a stability analysis. Repeated fits are needed to describe variation; a pair of fits can illustrate disagreement but cannot characterize its full pattern.
- Confusing a stable rank with a strong predictor. A predictor can rank consistently near the bottom, or consistently rank first among predictors whose importances are all close to zero.
- Ignoring the repeat scheme. Fixed-data refits assess algorithmic randomness; resampled-data refits also reflect changes in the observed cases. State which source of variation is represented.
Check Your Understanding
For each problem, calculate the requested stability summary before interpreting it.
- A predictor has ranks \(1,2,2,3\) across four fixed-data forest fits. Find its mean rank and top-two frequency. (Answer: Mean rank \(=2.0\); top-two frequency \(=3/4=0.75\).)
- Two no-tie rankings of four predictors have rank differences \(0,1,-1,0\). Calculate Spearman’s rank correlation. (Answer: \(\rho_s=0.8\).)
- A predictor's permutation importance values across five fits are \(0.3,0.1,-0.1,0.2,0.0\). Find its mean importance and state whether the average alone establishes a consistently positive contribution. (Answer: Mean importance \(=0.1\); no, the values include zero and a negative value, so the average alone does not establish consistently positive contribution.)
- Across six fits, correlated predictors \(U\) and \(V\) alternate between ranks 1 and 2, while their individual importance values are positive in every fit. What is the top-one frequency of each if each is rank 1 exactly three times, and what is a cautious interpretation? (Answer: \(3/6=0.5\) for each; their ranks are unstable individually, which may reflect shared information, but neither is shown to be unimportant.)
- A researcher refits a forest 20 times using different random seeds on the same data and reports that importance ranks vary. Which source of variation is represented, and which is not directly assessed? (Answer: Forest algorithmic randomness is represented; sampling variability from observing different cases is not directly assessed.)