Choosing mtry For A Random Forest

Why mtry Is a Tuning Choice

At each split in a random forest, the algorithm considers only a randomly selected subset of the available predictors. The number in that subset is called mtry. This creates a practical question: should each split see only a few predictors, or should it have access to most or all of them?

A larger candidate set gives a split more opportunity to use a predictor that produces a strong split. But when the same strong predictors are available repeatedly, different trees may make similar choices. A smaller candidate set makes the trees less likely to follow the same path, but can also leave a useful predictor out of consideration at a particular split. The best mtry balances these effects for the prediction task and data at hand; there is no universally best value.

This choice changes the set of split opportunities, not the number of predictors in the data, the bootstrap sample size, or the number of trees. To compare mtry values fairly, hold the training data and other fitting choices fixed, including the tree-growing rules and number of trees. The random subsets and bootstrap samples also introduce variation, so a small difference in one run should not automatically be treated as decisive.

Availability of Useful Predictors

A useful first calculation is the probability that at least one relevant predictor appears among the candidates at a split. Suppose there are \(p\) predictors in total, \(r\) of which are relevant to the split under consideration, and \(m\) candidates are sampled uniformly without replacement. The probability that none of the relevant predictors is included is the number of all-relevant-free subsets divided by the number of possible subsets.

This expression applies when \(m \leq p-r\); if \(m>p-r\), at least one relevant predictor must be included, so the probability is 1. The expected number of relevant candidates is a different quantity: it is \(mr/p\). That expectation is useful, but it does not give the probability of including at least one.

Tree Strength, Tree Correlation, and the Tradeoff

In this context, tree strength describes how well an individual tree predicts the outcome. It is a useful qualitative description, not a single universal score. With more predictors available at each split, a tree has more chances to choose a strong split, so increasing mtry often increases individual tree strength. It need not do so at every split or for every data set: additional candidates can be unhelpful, and trees also depend on their bootstrap samples and random choices.

Tree correlation describes how similarly different trees predict the same cases, especially in their prediction errors. Larger mtry often increases correlation because the same influential predictors are more frequently available, allowing trees to make similar splits. Smaller mtry can reduce this similarity by changing the available choices from split to split and tree to tree.

Averaging trees reduces the effect of their individual variation most when their errors are not too strongly correlated. But if a smaller mtry makes the trees substantially weaker, the reduction in correlation may not compensate. The practical goal is therefore not to minimize correlation by itself, nor to maximize the strength of each tree in isolation, but to obtain good ensemble prediction.

These are tendencies, not guarantees. If predictors are redundant, changing mtry may have little effect on useful split choices. If a small number of predictors dominate, a small mtry may often omit them, weakening trees. Conversely, when many predictors offer similar information, smaller mtry may give the forest varied trees without losing much split quality.

Using OOB Error to Compare Values

For an observation, a tree is out-of-bag (OOB) when that observation was not included in the tree’s bootstrap sample. Its OOB prediction is formed using trees for which it was OOB. Across the training observations, an OOB error measure summarizes how well those predictions perform. For classification, a common measure is the fraction of OOB classifications that are incorrect; for regression, a common measure is OOB mean squared error. The measure should match the task and the stated objective.

To tune mtry, fit forests at several candidate values and compare their OOB errors, keeping other settings fixed. Prefer a value with consistently good performance rather than assuming that error must improve monotonically as mtry changes. If the best values have very similar OOB error, consider the size and stability of the difference, computational cost, and the purpose of the analysis. If many choices are tried, the lowest observed OOB error can be optimistic because the tuning procedure has selected the best-looking result from several noisy estimates.

When Small Differences Need Careful Interpretation

OOB error is a practical internal estimate, not an error-free measurement of future performance. The fitted forests and their OOB predictions are affected by random sampling and feature selection. Comparing several mtry values also creates a selection effect: the minimum among noisy estimates may look better than the selected setting would perform on fresh data. When the choice matters, repeat the comparison under several random seeds or use a separate validation procedure to check whether the apparent advantage persists.

Repeated runs are particularly useful when candidate values have nearly equal errors. Compare both their average error and how much their errors vary across runs. A tiny difference in average error that changes direction across runs is weak evidence for a meaningful predictive advantage. When results are close, a less expensive setting may be reasonable, provided its performance is adequate for the purpose. This is a practical judgment, not a theorem that lower mtry is always preferable.

Common Exam Traps

• Confusing an expected count with a probability. For \(p=10\), \(r=2\), and mtry 2, \(mr/p=0.4\) is the expected number of relevant candidates. It is not the probability of at least one; that probability is \(1-\binom{8}{2}/\binom{10}{2}=17/45=0.3778\).
• Assuming larger mtry must lower forest error. A larger candidate set can strengthen individual trees, but can also increase their similarity. OOB performance must decide the comparison.
• Assuming the smallest mtry is best because it decorrelates trees. Lower correlation is useful only in combination with adequate tree strength.
• Comparing unlike fits. If the bootstrap scheme, tree settings, or number of trees changes along with mtry, the observed error difference cannot be attributed to mtry alone.
• Overinterpreting the minimum in a large search. Selecting the smallest of many OOB errors can capitalize on random variation; confirm important choices using repeated runs or validation data.

The useful workflow is to choose a reasonable range of mtry values, keep the rest of the fitting procedure controlled, and compare the appropriate OOB error. Interpret the result as evidence about predictive performance, not as a direct measurement of tree strength or correlation. This mtry tradeoff is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.