Tuning Mtry With Comparable Forest Fits

Why Compare mtry Values Systematically?

In a random forest, mtry is the number of candidate predictors considered at each split. Changing mtry changes the set of split candidates and can change both the individual trees and the forest’s predictions. The earlier material on random feature selection and out-of-bag predictions establishes these ideas; here the task is to compare a prespecified set of mtry values in a controlled way.

A useful comparison changes mtry and nothing else. If the number of trees, sampling method, tree-size controls, or class weighting also changes, a difference in out-of-bag (OOB) performance cannot be attributed to mtry alone. The workflow below is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: A comparable mtry tuning exercise fits one forest for each value in a prespecified candidate grid, using the same training data, response definition, and other modeling controls. It compares the resulting OOB performance using a consistent scoring rule and, when possible, the same cases.

For $p$ available predictors, a candidate mtry must be an allowed positive integer no greater than $p$. The grid should reflect the modeling question and be set before inspecting the scores. A grid might contain a few deliberately spaced values rather than every possible value. It is not necessary to assume that performance changes monotonically with mtry: the best value depends on the data and the forest settings.

Define the OOB Comparison Carefully

For a training case $i$, its OOB prediction is formed only from trees whose bootstrap sample did not include case $i$. A finite forest does not guarantee that every training case has at least one such tree. Define $A_m$ as the set of cases with at least one OOB tree for the forest fitted at mtry $m$. Only cases in $A_m$ have defined OOB predictions.

Definition: For regression, the OOB mean squared error at mtry $m$, calculated over cases with OOB predictions, is

\operatorname{OOB\ MSE}_m = \frac{1}{|A_m|} \sum_{i\in A_m} \left(y_i-\hat{y}^{\,\mathrm{OOB}}_{i,m}\right)^2.

For classification, the OOB misclassification rate is

\operatorname{OOB\ Error}_m = \frac{1}{|A_m|} \sum_{i\in A_m} I\!\left(\hat{y}^{\,\mathrm{OOB}}_{i,m}\ne y_i\right).

Here $I(\cdot)$ equals 1 when its condition is true and 0 otherwise. These formulas do not include cases with no OOB prediction, and their denominator is the number of eligible cases, not automatically the full training-set size. If $A_m$ is empty, the score is undefined.

The sets $A_m$ can differ across fitted forests. For the clearest comparison, calculate all candidate scores on the common eligible set $A^*=\bigcap_m A_m$, if that set is large enough to be informative. Otherwise, report each score with its coverage, $|A_m|$, and recognize that different case sets can affect the comparison. Do not silently count an uncovered case as correctly classified or assign it zero squared error.

Why Hold the Other Controls Fixed?

The OOB score is an empirical measure of prediction error for a particular fitting procedure. If two fits differ only in mtry, their scores provide evidence about the effect of changing that control under the chosen data and settings. If several controls change together, the comparison confounds their effects. Holding settings fixed does not make the scores exact or eliminate sampling variation, but it makes the intended comparison interpretable.

For example, fix the training rows and predictor columns, response type, number of trees, bootstrap or subsampling scheme, sample fraction, split rule, minimum node size, maximum depth, and any class weights or sampling controls. Also keep factor handling and missing-data processing consistent. A different seed can add another source of variation; using a stated seed makes a run reproducible, though resetting the seed does not guarantee identical tree-level randomness across different mtry values.

Set the comparison before fitting.
Choose the outcome, predictors, metric, candidate mtry grid, and all other forest controls. Confirm that every grid value is valid for the predictor count.

Fit one forest per candidate.
Use the same prepared training data and the same non-mtry settings for every fit. Record the seed and fitting configuration.

Check score coverage.
Confirm which cases have OOB predictions. Use a common eligible set when practical, or report coverage alongside each score.

Compare and communicate.
Select according to the prespecified metric and rule. Report the candidate scores and avoid presenting a small difference as conclusive when it may reflect forest randomness.

This procedure compares mtry while keeping controls fixed; it is not a search over all forest settings. The OOB score for a candidate is useful for comparison, but choosing the minimum from several scores can make that minimum look more favorable than its true predictive performance. The score is also calculated from the training data through OOB predictions, rather than from an independent test set. State that limitation rather than treating the selected value as guaranteed to perform best on new data.

Worked Example: Choosing mtry for a Regression Forest

A hypothetical actuary models claim severity using four predictors and compares mtry values 1, 2, and 4. Each forest has the same training data and all the same controls except mtry. The five cases in the common OOB-eligible set have the following residuals, defined as observed value minus OOB prediction. Select the mtry with the lowest common-set OOB MSE.

Method. Square each residual, sum the five squared residuals, and divide by five. Because the same five cases are used for all candidates, each score has the same denominator.

For mtry $=1$, the residuals are $1,-2,0,2,-1$. The squared residuals are $1,4,0,4,1$, so the sum is $1+4+0+4+1=10$. The OOB MSE is $10/5=2.00$.

For mtry $=2$, the residuals are $0.5,-1,1,0,-0.5$. The squared residuals are $0.25,1,1,0,0.25$, so the sum is $0.25+1+1+0+0.25=2.50$. The OOB MSE is $2.50/5=0.50$.

For mtry $=4$, the residuals are $1,1,-1,-1,0$. The squared residuals sum to $1+1+1+1+0=4$. The OOB MSE is $4/5=0.80$. The three scores are therefore 2.00, 0.50, and 0.80; mtry $=2$ has the lowest observed OOB MSE.

Check. Each score is a nonnegative average squared residual, and each uses the same five cases. A common mistake would be to divide one candidate’s sum by the full training-set size while using only five eligible cases for the others; that would not be a like-for-like comparison.

Implementing the Comparison in R

The following sketch uses ranger for a regression forest. Assume dat is a prepared training data frame, severity is the response, and the other columns are the $p=6$ predictors. The candidate grid is $1,2,4,6$, all within the allowed range. Set the number of trees and the other controls once, then vary only mtry.

In R, define mtry_grid = c(1, 2, 4, 6). Fit the forests with fits = lapply(mtry_grid, function(m) { set.seed(408); ranger(formula = severity ~ ., data = dat, num.trees = 500, mtry = m, min.node.size = 5, sample.fraction = 1, replace = TRUE, splitrule = "variance") }). Then retrieve the OOB scores with scores = sapply(fits, function(f) f$prediction.error). For a regression forest, prediction.error is the OOB mean squared error reported by ranger.

The seed, 500 trees, minimum node size, sampling fraction, replacement setting, and split rule are held fixed in this sketch. The number of trees is a fixed control here, not another candidate being compared. In an actual analysis, verify that the selected forest configuration and package settings match the intended response type and modeling choices. Keep the same rows and factor preparation across fits; otherwise the scores may reflect data changes as well as mtry.

Worked Example: Interpreting a Prespecified R Grid

A hypothetical severity analysis has six predictors and uses the R grid above. A compact diagnostic extract has five cases that are OOB-eligible for every candidate. The recorded residuals for those cases are: mtry 1, $3,3,0,0,0$; mtry 2, $2,2,2,0,0$; mtry 4, $3,2,0,0,0$; and mtry 6, $3,2,1,1,0$. Use this extract to interpret the comparison and state a suitable decision.

For mtry 1, the squared residual sum is $3^2+3^2+0+0+0=18$, giving MSE $18/5=3.60$. For mtry 2, it is $2^2+2^2+2^2+0+0=12$, giving MSE $12/5=2.40$.

For mtry 4, the sum is $3^2+2^2+0+0+0=13$, giving MSE $13/5=2.60$. For mtry 6, the sum is $3^2+2^2+1^2+1^2+0=15$, giving MSE $15/5=3.00$. Thus the lowest score in this extract is 2.40 at mtry 2.

Decision and check. On the stated criterion, mtry 2 is the observed winner. The next-lowest score is 2.60, only 0.20 higher, so report the scores and describe the result as a modest observed advantage, not proof that mtry 2 is universally superior. The calculation is internally consistent: all four sums use the same five cases and the same denominator. In a full fit, inspect OOB coverage as well as the reported score; the extract alone cannot establish coverage for every training case.

Classification Scores, Ties, and Common Traps

For classification, choose the scoring rule that matches the fitted forest and the modeling objective. With ordinary class predictions, OOB misclassification rate is the fraction of eligible cases whose OOB class differs from the observed class. If class weights or sampling controls are part of the analysis, hold them fixed across the mtry grid. A lower overall error does not, by itself, show that errors are equally acceptable across outcome classes.

When two candidates have the same score, the metric has not distinguished them. A practical tie-break, such as preferring a candidate with lower computational burden, should be stated in advance and should not be described as a predictive improvement. Similarly, small differences between scores should be interpreted cautiously: random forests involve random sampling and random predictor selection, so another random realization can change the ranking.

Common exam trap: Writing an OOB error sum over all $n$ training cases assumes every case has an OOB prediction. If that condition is not established, the formula is not defined for uncovered cases. Use only the eligible cases and divide by their count, or use the same common eligible set for every candidate. Report coverage so the reader can see how many cases contributed.

Another trap is changing controls while searching: for example, increasing mtry while also changing the sample fraction or minimum node size. That comparison may be useful for a separate modeling exercise, but it does not isolate mtry. A third trap is choosing the smallest reported number without checking whether the candidates used comparable cases or whether the difference is practically meaningful. A sound report names the grid, states what was held fixed, gives the OOB metric and its coverage, and describes the selection rule.

Check Your Understanding

For each problem, assume all stated forests use the same non-mtry settings unless the problem says otherwise.

A regression forest has $p=5$ predictors. Is the grid $1,3,5,6$ valid? State a corrected grid containing the valid candidates from that list. (Answer: No; mtry 6 exceeds $p=5$, so the corrected grid is 1, 3, 5.)
For one candidate, four cases have OOB residuals $1,-1,2,-2$, while a fifth case has no OOB prediction. Calculate the candidate’s MSE. (Answer: The squared residual sum is $1+1+4+4=10$; divide by 4 eligible cases, giving 2.50.)
Two classification forests are compared on the same 10 OOB-eligible cases. Candidate A makes 3 errors and candidate B makes 2. Calculate both rates and identify the lower-error candidate. (Answer: A has $3/10=0.3000$, B has $2/10=0.2000$; candidate B has the lower observed error.)
A five-predictor regression problem compares mtry values 1, 2, and 5. Their common-set OOB squared-error sums are 24, 18, and 20 on six cases. Calculate the three MSEs and choose by the stated metric. (Answer: The MSEs are $24/6=4.00$, $18/6=3.00$, and $20/6=3.33$; choose mtry 2 by lowest observed OOB MSE.)
Candidate A has OOB error $0.14$ on 100 eligible cases; candidate B has error $0.12$ on 75 eligible cases, and 25 cases lack an OOB prediction for B. What comparison issue should be reported, and what would improve comparability? (Answer: The scores use different case sets and coverage, so the difference may partly reflect which cases were scored; compare on the common eligible cases if feasible and report the coverage.)

What You'll Learn