Why a Forest Tuning Decision Needs More Than One Number
A classification forest can make different kinds of errors, and its apparent performance depends on how those errors are counted. A single OOB error rate is useful for comparing candidate settings, but it can hide poor detection of a less frequent class. Variable-importance measures add another kind of evidence: they describe how predictors contribute to the fitted forest under particular definitions, not whether a model is appropriate for a decision.
This worked case brings together three tasks: compare \(m_{\text{try}}\) settings using OOB predictions, inspect class-specific performance, and interpret permutation and impurity importance. The data throughout are hypothetical. They are fixed class counts and forest summaries, not observations from an insurer or a real portfolio.
The tuning quantity \(m_{\text{try}}\) is the number of candidate predictors made available when a tree considers a split. A smaller value adds more randomness to split selection; a larger value gives each split more opportunity to use strong predictors. In a comparison, keep other modeling choices fixed as far as possible, including the training data, tree count, sampling approach, and classification threshold. Compare predictions on the same evaluation observations, so differences in reported rates are not simply due to different denominators.
A Consistent Way to Read the Results
For two classes, call the class of interest positive. A true positive (TP) is a positive observation predicted positive; a false negative (FN) is a positive predicted negative. A true negative (TN) is a negative predicted negative; a false positive (FP) is a negative predicted positive. These counts give several different summaries:
Sensitivity is the fraction of actual positives detected, while specificity is the fraction of actual negatives correctly identified. Precision answers a different question: among observations predicted positive, what fraction are actually positive? In a setting with few positives, precision can be low even when specificity is high.
Worked Example: Selecting mtry for a Rare Claim Flag
A hypothetical insurer classifies whether a policy will have a specified claim flag during a future observation period. Among 250 OOB-evaluated policies, 50 have the flag and 200 do not. Three forests use \(m_{\text{try}}=2,3,\) or \(6\), with the same predictor set, tree count, and classification threshold. The OOB confusion matrices are:
| mtry | TN | FP | FN | TP |
|---|---|---|---|---|
| 2 | 171 | 29 | 14 | 36 |
| 3 | 180 | 20 | 18 | 32 |
| 6 | 184 | 16 | 24 | 26 |
Method. Calculate overall OOB error and class-specific measures for each setting. Then distinguish the setting with the lowest error from the setting with the lowest stated decision cost. The counts are the complete data for this calculation; no parametric outcome distribution, interest rate, or mortality assumption is involved.
For \(m_{\text{try}}=2\), there are \(29+14=43\) errors, so the OOB error is \(43/250=0.172\), or 17.2%. Sensitivity is \(36/50=0.720\). Specificity is \(171/200=0.855\). Precision is \(36/(36+29)=36/65=0.5538\), to four decimals.
For \(m_{\text{try}}=3\), there are \(20+18=38\) errors, giving \(38/250=0.152\), or 15.2%. Sensitivity is \(32/50=0.640\). Specificity is \(180/200=0.900\). Precision is \(32/(32+20)=32/52=0.6154\), to four decimals.
For \(m_{\text{try}}=6\), there are \(16+24=40\) errors, giving \(40/250=0.160\), or 16.0%. Sensitivity is \(26/50=0.520\). Specificity is \(184/200=0.920\). Precision is \(26/(26+16)=26/42=0.6190\), to four decimals.
By overall OOB error, \(m_{\text{try}}=3\) is best at 15.2%. But suppose a missed positive has a hypothetical decision cost of $800 and a false alarm has a cost of $100. The costs are \(14(800)+29(100)=\$14{,}100\) for \(m_{\text{try}}=2\), \(18(800)+20(100)=\$16{,}400\) for \(m_{\text{try}}=3\), and \(24(800)+16(100)=\$20{,}800\) for \(m_{\text{try}}=6\). On this stated cost criterion, \(m_{\text{try}}=2\) is preferred.
Check. The first row’s counts sum to \(171+29+14+36=250\); likewise, each other row sums to 250. The lowest-error choice and lowest-cost choice differ because the cost of a missed positive is much larger than the cost of a false alarm. A common mistake is to declare \(m_{\text{try}}=3\) best for every purpose just because it has the lowest OOB error.
Class-Specific Performance Can Change the Decision
The previous example shows why the positive class and the consequences of errors must be stated. In practice, the predicted class often results from comparing a forest’s estimated positive-class vote proportion with a threshold. A threshold of 0.5 is common, but it is not a universal requirement. Lowering the threshold will often identify more positives, while also producing more false positives; the actual changes depend on the observations’ predicted scores.
Worked Example: Comparing Two Classification Thresholds
A hypothetical forest produces OOB scores for 100 policies, 20 of which have a specified event and 80 of which do not. At threshold 0.50, the confusion counts are TN = 72, FP = 8, FN = 10, TP = 10. At threshold 0.35, they are TN = 66, FP = 14, FN = 6, TP = 14. A missed event costs $600 and a false alarm costs $80. Calculate the key rates and choose by the stated cost.
At threshold 0.50: The error count is \(8+10=18\), so error is \(18/100=0.18\). Sensitivity is \(10/20=0.50\); specificity is \(72/80=0.90\); precision is \(10/(10+8)=10/18=0.5556\), to four decimals. Cost is \(10(600)+8(80)=\$6{,}000+\$640=\$6{,}640\).
At threshold 0.35: The error count is \(14+6=20\), so error is \(20/100=0.20\). Sensitivity is \(14/20=0.70\); specificity is \(66/80=0.825\); precision is \(14/(14+14)=0.50\). Cost is \(6(600)+14(80)=\$3{,}600+\$1{,}120=\$4{,}720\).
The 0.35 threshold has a higher error rate but a lower stated cost, because it reduces false negatives by four and adds six false positives. The cost change is \(4(600)-6(80)=\$2{,}400-\$480=\$1{,}920\) in savings, agreeing with \(\$6{,}640-\$4{,}720=\$1{,}920\). The preferred threshold under this cost rule is 0.35. Counts are given directly; no probability distribution for the scores and no interest or mortality assumption is needed.
Common mistake avoided: Choosing the threshold with the smaller total error would select 0.50, despite the problem specifying unequal error costs. Also avoid calling precision “sensitivity”: precision has predicted positives in its denominator, while sensitivity has actual positives.
Interpreting Two Importance Rankings
Permutation importance asks how much predictive performance deteriorates when a predictor’s values are randomly rearranged, with the fitted forest otherwise fixed. For an OOB permutation calculation, the rearrangement is made for the relevant OOB observations and the change in their OOB performance is recorded. A larger increase in error indicates a stronger reliance on that predictor under this particular model and evaluation procedure. It is not a causal effect.
Mean decrease in impurity (MDI) instead aggregates the reductions in node impurity attributed to splits on a predictor across the forest, usually averaging or summing those reductions according to the implementation. The scales of permutation importance and MDI are different. Their numerical values should not be compared directly as though they were the same unit.
Worked Example: Reconciling Different Importance Rankings
A hypothetical classification forest has 300 OOB observations. Its baseline OOB error is 45 misclassifications. The analyst permutes each predictor separately and records the following errors. For MDI, the table gives the impurity-reduction score for each of four trees; the forest score is the mean across those four trees.
| Predictor | Errors after permutation | MDI scores across four trees |
|---|---|---|
| Prior claim count | 66 | 8, 7, 9, 8 |
| Annual mileage | 57 | 6, 5, 7, 6 |
| Sales channel | 48 | 10, 9, 8, 9 |
Method. First compute the baseline error rate, then each permuted error rate and its increase in percentage points. Separately calculate each MDI mean and rank the predictors within each measure. Treat the scores as hypothetical forest summaries, not evidence about actual policyholders.
The baseline error rate is \(45/300=0.15\), or 15%. For prior claim count, the permuted error rate is \(66/300=0.22\), or 22%. Its increase is \(0.22-0.15=0.07\), which is 7 percentage points. For annual mileage, the rate is \(57/300=0.19\), giving an increase of \(0.19-0.15=0.04\), or 4 percentage points. For sales channel, the rate is \(48/300=0.16\), giving an increase of \(0.16-0.15=0.01\), or 1 percentage point.
The MDI means are \((8+7+9+8)/4=32/4=8\) for prior claim count, \((6+5+7+6)/4=24/4=6\) for annual mileage, and \((10+9+8+9)/4=36/4=9\) for sales channel. Thus permutation importance ranks prior claim count first, mileage second, and channel third. MDI ranks channel first, prior claim count second, and mileage third.
The disagreement is not an arithmetic contradiction: the measures assess different properties. A predictor can be selected in impurity-reducing splits yet cause only a small drop in OOB performance when permuted. The appropriate interpretation is that the forest’s importance ordering depends on the chosen measure; it is not justified to average the rankings or claim that either one proves a predictor causes the event. The baseline and permuted error counts are all out of 300, so their comparisons use a common denominator. There is no assumed parametric distribution, interest rate, or mortality model in this example.
Common mistake avoided: Reporting the MDI score of 9 as “nine percentage points of predictive importance” is incorrect. It is a score on the impurity measure’s scale, not an error-rate change.
Practical Interpretation and Checks
An exam-ready analysis should state the selection rule before naming a winning forest. If the criterion is minimum OOB error, compare the error rates on a common evaluation set. If missing positives is especially costly, also examine sensitivity and the specified cost function. When predicted probabilities or vote proportions are thresholded, report the threshold; otherwise, performance comparisons may not be interpretable.
Importance rankings should be read as diagnostics of a fitted forest, not as a standalone feature-selection instruction. Permutation importance can be affected by how predictors share information, and MDI can favor predictors that offer more possible split points. Rankings can also change across samples or forest fits. If measures disagree, describe the disagreement and the measure used rather than forcing a single definitive ranking.
Check Your Understanding
For each hypothetical case, calculate the requested quantities before checking the final answer.
- A forest has TN = 144, FP = 16, FN = 12, and TP = 28 on 200 OOB cases. Find its error rate, sensitivity, and specificity. (Answer: error \(=28/200=0.14\); sensitivity \(=28/40=0.70\); specificity \(=144/160=0.90\).)
- Two candidate forests have OOB counts \((\mathrm{FP},\mathrm{FN})=(9,21)\) and \((18,12)\), respectively, on the same 300 cases. Missed positives cost $500 and false positives cost $100. Find each total cost and choose by cost. (Answer: first \(=9(100)+21(500)=\$11{,}400\); second \(=18(100)+12(500)=\$7{,}800\); choose the second.)
- There are 40 actual positives. A threshold produces TP = 30, FN = 10, FP = 20, and TN = 140. Calculate sensitivity, precision, and total error rate. (Answer: sensitivity \(=30/40=0.75\); precision \(=30/50=0.60\); error \(=30/200=0.15\).)
- Baseline OOB error is 36/240. Permuting a predictor gives 54 errors. Find the increase in error in percentage points. (Answer: baseline \(=15\%\), permuted \(=22.5\%\), increase \(=7.5\) percentage points.)
- Two predictors have four-tree MDI scores of 4, 6, 5, 5 and 7, 6, 8, 7. Their permutation error increases are 5 and 3 percentage points, respectively. Rank them separately by mean MDI and permutation importance, and state one interpretation limit. (Answer: mean MDI values are 5 and 7, so the second predictor ranks higher by MDI; the first ranks higher by permutation importance. The rankings measure different aspects of the fitted forest and establish no causal effect.)