Why Forest Results Can Disagree
A random forest can have a stable out-of-bag (OOB) error estimate and still produce importance rankings that appear inconsistent. One predictor may have the largest impurity importance but a small permutation importance; another may have a large permutation importance despite rarely appearing near the top of trees. These results are not automatically contradictory. They summarize different aspects of the fitted forest.
OOB performance describes how well the forest predicts observations excluded from the trees used to predict them. Permutation importance measures how much a chosen prediction loss changes when a predictor's values are disrupted. Impurity importance summarizes the reductions in node impurity attributed to splits on each predictor. Their units, interpretations, and sensitivities differ. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The loss must be named before interpreting permutation importance. For classification, it might be misclassification error or a probability-based loss; for regression, it might be mean squared error (MSE). A positive increase means prediction performance worsened after permutation under that loss. A near-zero increase means little measured deterioration in that particular permutation procedure; it does not prove that the predictor has no relationship with the outcome.
Impurity importance is different. It is based on the training splits, not on how much a held-out prediction loss worsens after disrupting a predictor. Its raw scale depends on the impurity measure, tree construction, and aggregation. A value of 20 for impurity importance cannot be directly compared with a permutation increase of 0.05. Even two forests' impurity values may not be comparable if their fitting settings differ.
What the Permutation Calculation Measures
Let \(L_{\mathrm{OOB}}\) be the average OOB loss from the original predictions, and let \(L_{\mathrm{OOB},j}^{\mathrm{perm}}\) be the average loss after permuting predictor \(X_j\) and recomputing the relevant OOB predictions. A common definition of permutation importance is
Some implementations repeat the permutation and average the increases; some report a standardized quantity or a differently scaled summary. Always check the software's definition before comparing values. The basic interpretation remains conditional on the chosen loss and permutation procedure.
The reason for the subtraction is direct: it compares the model's loss when the predictor retains its observed information with the loss when that information is disrupted. If the predictor matters to the fitted forest's predictions, the disrupted predictions will tend to be worse. A permutation does not refit the forest; it changes predictor values for the importance calculation while holding the fitted model fixed.
Here \(N\) is the number of observations, \(y_i\) is the observed response, \(\hat{f}_{i,\mathrm{OOB}}\) is the OOB prediction, and \(\ell\) is the selected loss function. Subtracting the first average loss from the second gives the average loss change attributable to disrupting \(X_j\) in this fitted forest. This is a measure of predictive reliance under the procedure, not a causal effect.
Worked Example: A Strong Permutation Signal but Lower Impurity Importance
A hypothetical binary-outcome forest has OOB classifications for 200 observations. The original forest misclassifies 16. Permuting predictor A results in 31 misclassifications, while permuting predictor B results in 22. The forest reports raw impurity importance of 18 for A and 27 for B. Diagnose the disagreement using OOB misclassification error as the loss.
Method. Compute the original OOB error and each permuted OOB error, then subtract the original error. Treat the impurity values as a separate split-based summary, not as loss increases.
Original error is \(16/200=0.080\). After permuting A, the error is \(31/200=0.155\), so \(I_A^{\mathrm{perm}}=0.155-0.080=0.075\). After permuting B, the error is \(22/200=0.110\), so \(I_B^{\mathrm{perm}}=0.110-0.080=0.030\).
Permutation importance ranks A above B: disrupting A increases error by 0.075, compared with 0.030 for B. Impurity importance ranks B above A: 27 is greater than 18. The two rankings need not agree because impurity importance counts split improvements in the fitted trees, while permutation importance assesses deterioration in OOB prediction after disrupting a predictor.
A useful next check is whether B has many possible split points or factor levels. Predictors with more split opportunities can receive larger impurity importance under some tree-building procedures, even when their permutation importance is modest. The available results do not establish that this is the cause; predictor structure and fitting choices need inspection. The common mistake is to call the impurity value 27 “more predictive” than A's permutation value 0.075. Those values are on different scales and answer different questions.
Diagnosing Predictor Structure
Before interpreting a disagreement as evidence that one measure is wrong, inspect how the predictors relate to one another and how each is represented in the data. Correlated predictors are an important case. A forest may use either of two similar predictors to make useful splits. Permuting one alone can leave the other available as a substitute, resulting in a modest individual importance for both even when the information they jointly carry is useful.
The pattern can also go in the other direction. If the forest happens to rely heavily on one member of a correlated group, permuting that predictor may cause a large loss increase, even though another predictor contains similar information. Thus, an individual ranking is not a ranking of unique, independent contributions. Conditional permutation procedures can help investigate importance while accounting for predictor structure, but their results still depend on their definitions and assumptions.
Other useful checks include:
- Whether a predictor is continuous, discrete, or a factor with many levels, and how many distinct split opportunities it offers.
- Whether rare factor levels or sparse combinations are driving splits.
- Whether predictors have missing values or encoding differences that affect the fitted trees.
- Whether an identifier, leakage variable, or near-duplicate predictor may be carrying information in an unintended way.
- Whether the importance measure was calculated on the same observations and with the same loss for every predictor.
Worked Example: Correlated Predictors and Joint Permutation
In a hypothetical regression problem, a forest has OOB MSE 9.6. Predictors A and B measure closely related aspects of an asset's condition. Permuting A alone gives MSE 16.8; permuting B alone gives MSE 15.9; jointly permuting both gives MSE 22.4. Calculate the loss increases and explain what the pattern suggests.
Method. Subtract the same original OOB MSE from each permuted MSE. The joint permutation is a separate diagnostic; its increase is not defined as the sum of the two individual increases.
For A alone, the increase is \(16.8-9.6=7.2\). For B alone, it is \(15.9-9.6=6.3\). For the joint permutation, it is \(22.4-9.6=12.8\).
Both individual increases are substantial, and disrupting both produces a larger increase than disrupting either alone. This is consistent with the pair carrying useful predictive information. It does not imply that their individual contributions are independent: the joint increase of 12.8 is not the sum \(7.2+6.3=13.5\). A and B may overlap in the information they provide, and the fitted forest's predictions are nonlinear combinations of splits.
The appropriate conclusion is that individual importance should be interpreted in light of the predictors' relationship. A low individual score would not by itself establish that a correlated predictor is unhelpful. The common mistake is to add the two individual importance increases and treat the sum as the pair's joint importance.
Check Whether the Forests Are Comparable
An importance disagreement can arise because the forests being compared are not actually comparable. Before comparing rankings across fits, check that they use the same response definition, analysis rows, predictor coding, loss, class weighting or sampling scheme, and importance calculation. Also record settings such as the number of candidate predictors considered at each split (\(mtry\)), tree-size controls, and bootstrap sampling or subsampling choices. These settings can change which predictors are available or selected and can therefore change both OOB performance and importance.
A comparison across different fitting choices can be informative, but it answers a different question from a comparison of two importance measures from the same fitted forest. If a fit changes its sampling scheme or class weights, a changed ranking may reflect changed training emphasis, not a simple correction of the first ranking. Similarly, a different OOB error can reflect a different fitting procedure, an OOB sample difference, or a changed prediction objective.
Performance itself also depends on the metric. Ordinary misclassification error weights all observations equally. When outcomes are imbalanced, it can be low even when predictions identify the less common class poorly. If the objective values sensitivity and specificity more evenly, balanced accuracy may be more appropriate. The metric should follow the stated decision objective rather than being chosen to make a preferred model look better.
Worked Example: OOB Error and Balanced Accuracy Rank Fits Differently
Two hypothetical forests are fitted to the same 200 observations: 30 positive cases and 170 negative cases. Forest A has 18 true positives, 12 false negatives, 156 true negatives, and 14 false positives. Forest B has 24 true positives, 6 false negatives, 144 true negatives, and 26 false positives. Compare their OOB error and balanced accuracy. For a further diagnostic, after permuting predictor X in forest A, the confusion counts are 14 true positives, 16 false negatives, 150 true negatives, and 20 false positives; after permuting predictor Z they are 17, 13, 146, and 24, respectively.
Method. Calculate OOB error as total errors divided by 200. Calculate sensitivity as true positives divided by the 30 actual positives, specificity as true negatives divided by the 170 actual negatives, and balanced accuracy as the average of sensitivity and specificity.
For forest A, errors are \(12+14=26\), so OOB error is \(26/200=0.130\). Its sensitivity is \(18/30=0.600\), and its specificity is \(156/170\approx0.9176\). Its balanced accuracy is \((0.600+0.9176)/2\approx0.7588\).
For forest B, errors are \(6+26=32\), so OOB error is \(32/200=0.160\). Its sensitivity is \(24/30=0.800\), and its specificity is \(144/170\approx0.8471\). Its balanced accuracy is \((0.800+0.8471)/2\approx0.8235\). Forest A has lower ordinary error, but forest B has higher balanced accuracy. Neither ranking is arithmetically inconsistent; the metrics emphasize different performance properties.
For forest A after permuting X, the error is \((16+20)/200=0.180\), an increase of \(0.180-0.130=0.050\). Sensitivity becomes \(14/30\approx0.4667\), and specificity becomes \(150/170\approx0.8824\), giving balanced accuracy of about 0.6745. The balanced-accuracy decrease is about \(0.7588-0.6745=0.0843\).
After permuting Z, the error is \((13+24)/200=0.185\), an increase of \(0.185-0.130=0.055\). Sensitivity becomes \(17/30\approx0.5667\), and specificity becomes \(146/170\approx0.8588\), giving balanced accuracy of about 0.7127. The balanced-accuracy decrease is about \(0.7588-0.7127=0.0461\).
Under ordinary error, Z has the larger permutation increase, 0.055 versus 0.050. Under balanced accuracy, X has the larger decrease, about 0.0843 versus 0.0461. The importance ranking depends on the loss, just as model performance rankings do. The common mistake is to call one ranking universally correct without first identifying the prediction objective.
A Practical Diagnostic Sequence
Use a consistent sequence when OOB performance and importance summaries seem to conflict. This helps separate a genuine modeling question from a comparison made on incompatible scales or under different assumptions.
Record the exact OOB metric, permutation loss and scaling, impurity measure, and whether any values are normalized.
Check that the same response, rows, predictors, factor coding, sampling, class weights, and relevant forest controls are being compared.
Look for correlated predictors, many-level factors, rare levels, missingness patterns, and potential leakage. Consider whether a predictor group is more informative than any member alone.
For imbalanced classification, inspect class-specific performance as well as overall error. For regression, verify that the reported loss matches the intended cost of prediction errors.
Describe what the fitted forest's OOB predictions and chosen importance measure show. Do not convert a ranking into a causal claim or a claim about unique predictor value.
Common Interpretation Traps
- Comparing unlike scales. Impurity totals and permutation loss increases are not interchangeable numbers. Compare rankings within a measure only when the definitions and fits are compatible.
- Treating importance as model performance. A predictor can have high importance in a forest whose OOB performance is poor. Importance does not tell whether the forest predicts well overall.
- Assuming a low individual importance means no useful information. Correlated predictors can substitute for one another. Inspect their structure and, where suitable, assess the group or use a conditional procedure.
- Ignoring the evaluation metric. A variable ranking under overall error can differ from one under a metric that gives more weight to a minority class.
- Comparing changed fits as though only one detail changed. If sampling, weights, controls, or data rows differ, describe those differences before attributing a changed ranking to predictor behavior.
Check Your Understanding
For each problem, identify the calculation and the interpretation that the stated information supports.
- A hypothetical forest has 250 OOB classifications and makes 20 errors. Permuting predictor R produces 37 errors. Calculate the original OOB error and R's permutation importance using misclassification error. (Answer: OOB error \(=20/250=0.080\); permuted error \(=37/250=0.148\); importance \(=0.068\).)
- A regression forest has OOB MSE 12.5. Permuting predictor S gives MSE 17.0. Calculate the permutation importance on the MSE scale and state what it measures. (Answer: \(17.0-12.5=4.5\) MSE units; the OOB MSE increase after disrupting S.)
- In one forest, predictors U and V have impurity importances 42 and 35. Their permutation increases in OOB loss are 1.2 and 4.8, respectively. State each ranking and explain why the results are not contradictory. (Answer: impurity ranks U above V; permutation ranks V above U; they summarize split reductions and OOB loss changes on different scales.)
- A classification forest has 40 actual positives and 160 actual negatives. Its OOB predictions give 28 true positives and 12 false negatives, plus 136 true negatives and 24 false positives. Calculate error, sensitivity, specificity, and balanced accuracy. (Answer: error \(=36/200=0.180\); sensitivity \(=28/40=0.700\); specificity \(=136/160=0.850\); balanced accuracy \(=(0.700+0.850)/2=0.775\).)
- Two highly correlated predictors each have small individual permutation importance, but jointly permuting them increases OOB loss substantially. Give the most defensible interpretation. (Answer: the pair may carry useful predictive information that overlaps; the small individual values do not establish that either predictor or the pair is uninformative.)