Calculating OOB Permutation Importance

Why Permute a Predictor?

A random forest can predict well by combining many trees, but prediction accuracy alone does not show how much the forest relies on a particular predictor. OOB permutation importance estimates how much predictive performance worsens when the association between one predictor and the response is disrupted. The comparison is made on observations that were out of bag for the trees used to predict them.

The essential discipline is to compare like with like. Baseline and permuted predictions must use the same evaluation observations, the same forest, and the same performance metric. If any of these changes, the difference cannot be attributed cleanly to shuffling the predictor.

Definition: For a predictor $X_j$, OOB permutation importance is the change in a chosen OOB error measure after the observed values of $X_j$ are shuffled among the relevant OOB evaluation cases, with the fitted forest and all other predictor values held fixed. Define the change as $$ I_j = E_{j,\mathrm{perm}} - E_{\mathrm{base}}, $$ where $E_{\mathrm{base}}$ is baseline OOB error and $E_{j,\mathrm{perm}}$ is OOB error after permuting $X_j$. For an error measure where smaller is better, positive importance means performance worsened after permutation.

Constructing a Fair OOB Comparison

In a bootstrap forest, each tree is fitted using a bootstrap sample. An observation not selected in a tree's bootstrap sample is out of bag for that tree and can be evaluated by that tree without having been used to fit it. For observation $i$, its OOB prediction combines predictions from trees for which $i$ was OOB. The number of such trees can vary by observation.

For a particular predictor, a standard tree-level calculation proceeds as follows. For each tree, retain its OOB evaluation cases, calculate baseline predictions, then shuffle the selected predictor among those same cases and predict again with that tree unchanged. Aggregate predictions over the same OOB trees for each observation, and calculate both errors over the same set of observations. This preserves the comparison even though OOB coverage differs among observations.

Set the evaluation rule.
Specify the OOB observations and how their predictions are aggregated. Keep this rule fixed for the baseline and permuted calculations.

Calculate baseline error.
Use each observation's OOB prediction and the selected metric, such as mean squared error for regression or misclassification rate for classification.

Shuffle one predictor.
Within the relevant OOB evaluation cases, randomly reassign the values of $X_j$. Do not shuffle the response or change the other predictors.

Recalculate and compare.
Predict with the same fitted trees, cases, and metric. Subtract baseline error from permuted error.

For regression, if $y_i$ is the response and $\hat y_i^{\,\mathrm{base}}$ and $\hat y_i^{\,\mathrm{perm}}$ are the two OOB predictions for the same $n$ evaluation cases, the mean squared errors are

\mathrm{MSE}_{\mathrm{base}} = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i^{\,\mathrm{base}})^2, \qquad \mathrm{MSE}_{\mathrm{perm}} = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i^{\,\mathrm{perm}})^2, \qquad I_j = \mathrm{MSE}_{\mathrm{perm}}-\mathrm{MSE}_{\mathrm{base}}.

For classification, the chosen metric might be the misclassification rate, the proportion of evaluation cases assigned the wrong class. The same class-decision rule must be used in both calculations. Other metrics, such as log loss, are also possible, but the baseline and permuted values must be calculated using the same metric and the same probability convention.

Why the Difference Measures Predictive Reliance

The method has a simple justification. The baseline predictions use the predictor values in their observed alignment with the responses and with the other predictors. Shuffling $X_j$ breaks that alignment while retaining the shuffled values themselves. The fitted trees are not changed; only the values supplied for this predictor at prediction time are changed. Thus, the difference in error measures how much predictive performance depends on the information carried by $X_j$ in this fitted forest and evaluation sample.

This is a predictive diagnostic, not a causal effect. A large positive value indicates that disrupting this predictor worsened the selected OOB metric. A value near zero indicates little measured deterioration in this calculation. A negative value can occur from finite-sample variation: the shuffled version happened to perform better on these cases. It does not establish that the predictor is harmful in general.

Worked Example: Regression MSE Increase

A hypothetical forest is evaluated on four OOB cases. The responses are $2,4,6,8$. Its baseline predictions are $3,3,7,7$, and after permuting one predictor its predictions are $2,6,4,10$. Calculate the baseline MSE, permuted MSE, and OOB permutation importance. Use MSE as the metric throughout.

Method. Calculate squared residuals on the same four cases in each condition, divide each sum by four, and subtract baseline MSE from permuted MSE.

Baseline calculation. The residuals $y_i-\hat y_i^{\,\mathrm{base}}$ are $2-3=-1$, $4-3=1$, $6-7=-1$, and $8-7=1$. Their squares are $1,1,1,1$, so their sum is $4$, and

\mathrm{MSE}_{\mathrm{base}} = \frac{1+1+1+1}{4} = 1.

Permuted calculation. The residuals are $2-2=0$, $4-6=-2$, $6-4=2$, and $8-10=-2$. Their squares are $0,4,4,4$, with sum $12$. Therefore,

\mathrm{MSE}_{\mathrm{perm}} = \frac{0+4+4+4}{4} = 3.

Importance and check. The increase is $3-1=2$ squared response units. The permuted error is larger than baseline error, as expected for a predictor whose shuffle worsens these predictions. The common mistake is to report the permuted MSE, $3$, as the importance; importance is the difference, $2$.

Classification and Repeated Permutations

For classification with misclassification rate, let $m_{\mathrm{base}}$ be the number of incorrect baseline classifications and $m_{\mathrm{perm}}$ the number after shuffling, among the same $n$ cases. Then

I_j = \frac{m_{\mathrm{perm}}}{n} - \frac{m_{\mathrm{base}}}{n}.

The result is an increase in error rate, not an increase in accuracy. If a software output instead reports accuracy, where larger is better, the direction of subtraction must be stated explicitly; an accuracy decrease is baseline accuracy minus permuted accuracy.

A shuffle is random, so a single permutation can produce a noisy estimate. One can repeat the shuffle $B$ times, calculate an error increase for each, and average the increases:

\bar I_j = \frac{1}{B}\sum_{b=1}^{B} \left(E_{j,\mathrm{perm}}^{(b)}-E_{\mathrm{base}}\right).

The baseline error stays fixed when the baseline predictions stay fixed; each replicate changes the shuffled predictor assignment. Reporting the number of permutations and the metric helps make the importance interpretable.

Worked Example: Classification Error-Rate Increase

A hypothetical classifier is evaluated on six OOB cases with true classes $A,A,A,B,B,B$. Its baseline predicted classes are $A,A,A,B,B,A$. After shuffling a predictor, the predicted classes are $A,B,A,B,A,B$. Calculate both misclassification rates and the importance.

Baseline. The first five predictions are correct and the sixth is incorrect, so there is $1$ error among $6$ cases:

E_{\mathrm{base}} = \frac{1}{6} = 0.1667.

Permuted. Comparing the shuffled predictions with the true classes, cases 2 and 5 are incorrect, while cases 1, 3, 4, and 6 are correct. Thus there are $2$ errors:

E_{\mathrm{perm}} = \frac{2}{6} = 0.3333.

Importance and check. The error-rate increase is $2/6-1/6=1/6=0.1667$, or 16.67 percentage points. The calculation uses the same six cases and the same classification rule in both conditions. The common mistake is to subtract error rates in the reverse order, which would yield a negative value despite the observed deterioration.

Worked Example: Averaging Results Across Shuffles

For a hypothetical regression forest, the baseline OOB MSE on a fixed set of three cases is $1$. Three independent shuffles of the same predictor give permuted OOB MSE values $4/3$, $4$, and $1$. Find the importance for each shuffle and the average permutation importance.

First shuffle. Its increase is $4/3-1=1/3$.

Second shuffle. Its increase is $4-1=3$.

Third shuffle. Its increase is $1-1=0$. The shuffled predictor happened not to worsen error on this replicate.

Average and check. The mean increase is

\bar I_j = \frac{1/3+3+0}{3} = \frac{10/3}{3} = \frac{10}{9} \approx 1.111.

The three shuffles differ because each creates a different reassignment of predictor values. The mean is positive, even though one shuffle has zero increase. The common mistake is to average the three permuted MSE values and call that average the importance; the baseline MSE must first be subtracted for each replicate, or equivalently subtracted from the average permuted MSE.

Common Traps and Interpretation

Changing the evaluation cases. Excluding cases after permutation, or evaluating the baseline and shuffled predictions on different OOB subsets, makes the difference reflect both the shuffle and the changed sample. Keep the evaluation set fixed.
Changing the metric. Do not compare baseline MSE with permuted absolute error, or baseline error rate with permuted accuracy. Calculate one metric consistently.
Retraining after the shuffle. Retraining changes the fitted trees as well as the predictor alignment. Permutation importance evaluates the existing forest with shuffled predictor values; it does not measure the result of fitting a new forest.
Shuffling the response or every predictor. Only the predictor under study is shuffled. The response and remaining predictor values stay attached to their original evaluation cases.
Confusing importance with a causal effect. The result describes a change in predictive error under a specified procedure. It does not quantify what would happen if an intervention changed the predictor.

In practice, the reported value is meaningful only alongside its metric and evaluation procedure. For example, an importance of $0.02$ means an increase of $0.02$ in the selected error measure, not a universal scale of importance. A positive value is evidence of deterioration in that OOB comparison; a small or negative value should be read with the randomness of OOB sampling and permutation in mind.

Key takeaway: OOB permutation importance is a paired performance comparison: hold the forest, OOB evaluation cases, and metric fixed; shuffle one predictor; then subtract baseline error from permuted error. For error metrics, a positive difference means the shuffle made predictions worse.

Check Your Understanding

For each problem, use the same evaluation cases and metric before and after permutation.

A regression forest has responses $1,3,5$, baseline predictions $2,2,4$, and permuted predictions $1,5,5$. Find both MSE values and the MSE increase. (Answer: Baseline MSE $=1$; permuted MSE $=4/3$; importance $=1/3$.)
A classifier makes 2 errors among 8 OOB cases before permutation and 5 errors among the same 8 cases afterward. Find the two error rates and importance. (Answer: Baseline error rate $=0.2500$; permuted error rate $=0.6250$; importance $=0.3750$.)
On three fixed regression cases, baseline predictions have absolute errors $1,1,1$, while permuted predictions have absolute errors $2,2,2$. Using mean absolute error, calculate importance. (Answer: Baseline MAE $=1$; permuted MAE $=2$; importance $=1$.)
A classifier's baseline mean log loss is $0.42$, and its mean log loss after predictor permutation is $0.57$, using the same cases and probability-based loss definition. What is the importance and its interpretation? (Answer: $0.15$ log-loss units; the shuffled predictor worsened the measured OOB performance.)
A proposed calculation uses baseline OOB error on 40 cases but permuted error on 36 cases because four cases lack a permuted prediction. Is the difference a fair permutation-importance estimate under the fixed-case definition? State the needed correction. (Answer: No; use a common evaluation set with both baseline and permuted predictions for every included case, or otherwise define and apply a consistent missing-prediction rule to both conditions.)

What You'll Learn