Choosing Mtry For Classification And Regression

Why the Number of Candidate Predictors Matters

At each split in a random forest, a tree considers only a randomly selected subset of the available predictors. The number in that subset is called mtry. It controls how many opportunities a split has to use a strong predictor, and how much randomness remains between trees. Choosing mtry is therefore a balance: a larger candidate set may make stronger splits easier to find, while a smaller one generally makes the trees’ split decisions less alike.

The right balance depends on the data and the prediction task. Common classification and regression defaults give useful starting points, but they do not guarantee the best forest for a particular problem. This tutorial develops the trade-off and the probability calculations that help explain it. The discussion is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

A candidate predictor is not the same thing as a candidate split point. For example, a numerical predictor might offer several possible cut points. The mtry selection determines whether that predictor is considered at all; the tree’s split-search procedure then assesses its possible splits. For classification trees, the assessment commonly uses an impurity measure such as Gini impurity. For regression trees, it commonly uses reduction in squared error.

Unless noted otherwise, probability calculations below assume that the \(m\) candidates are selected uniformly without replacement from the \(p\) available predictors. They also assume that the same set of \(p\) predictors is eligible at the node and that the predictors called useful are fixed for the calculation. These assumptions make the candidate set a simple random sample; implementations or data conditions that change eligibility can alter the calculation.

Common Starting Rules

Two widely used rules of thumb are:

• Classification: start with \(m\) near \(\sqrt{p}\).
• Regression: start with \(m\) near \(p/3\).

These are starting conventions, not mathematical optima. The software must use an integer between 1 and \(p\), so it has to apply an integer rule to a non-integer result. Packages and versions can differ in their defaults and rounding conventions. In practice, check the software’s documented default and the fitted model’s reported setting rather than assuming that every implementation makes the same choice.

Increasing \(m\) makes it more likely that a strong predictor is available to a node. It also gives the tree a larger pool from which to choose its split. Reducing \(m\) adds randomization: even two trees seeing similar observations may receive different candidate predictors and therefore make different split decisions. This can reduce similarity among trees, although it can also make a particular tree’s split weaker. Neither effect alone determines the forest’s predictive performance.

Probability of Including a Useful Predictor

More generally, suppose \(k\) of the \(p\) predictors are useful at a node, and \(m\) predictors are sampled without replacement. “Useful” here means useful for the split opportunity under consideration; it does not imply that the predictor is useful at every node or that it will be chosen as the final split.

The numerator counts candidate sets containing only predictors from the \(p-k\) non-useful predictors. The denominator counts all candidate sets of size \(m\). Their ratio is the probability of getting a set with no useful predictor; subtracting from 1 gives the probability of the complementary event.

This formula also shows why coverage rises with \(m\): increasing the number drawn makes it harder to form a candidate set consisting entirely of non-useful predictors. But coverage is not the only consideration. If one predictor is exceptionally strong, a small \(m\) may often omit it. If many predictors carry similar information, a smaller \(m\) may still provide useful candidates while introducing more variation in the trees’ choices.

How mtry Changes Split Strength and Tree Diversity

At a given node, imagine listing the split improvements available from all eligible predictors, with each predictor’s best valid split represented by its improvement under the tree’s criterion. If the same data, eligibility rules, and split-search method are held fixed, considering more predictors cannot reduce the best improvement available: the larger candidate set contains more options to compare. This is an opportunity statement, not a claim that every realized forest with larger \(m\) will perform better.

The justification is direct. Let \(C_1\) be a candidate set contained in a larger candidate set \(C_2\). The maximum improvement among predictors in \(C_1\) is no greater than the maximum among predictors in \(C_2\), because every option in \(C_1\) is also available in \(C_2\). Random candidate sets of different sizes are not necessarily nested in an actual fit, so this comparison explains the mechanism rather than proving a point-by-point ordering of two fitted forests.

A larger \(m\) also makes it more likely that a high-value predictor is repeatedly available. If one predictor dominates, many trees may select it near the top of their structure. Smaller \(m\) makes its omission more likely at any given node and gives other predictors more opportunities to shape the trees. This can reduce tree-to-tree similarity, a consideration connected to the earlier material on tree correlation and ensemble variance. However, a smaller candidate set can also make useful splits less accessible, especially when only a few predictors carry signal.

Choosing and Interpreting a Starting Value

The classification square-root rule and regression one-third rule are practical defaults, not consequences of a theorem that identifies the best value. A useful interpretation is: begin with a conventional candidate count, then assess whether that setting provides an adequate balance between useful splits and tree diversity for the particular prediction problem. The best balance depends on the number of predictors, how concentrated the predictive signal is, relationships among predictors, and the split criteria and other forest settings.

A larger candidate count is especially attractive when important signal is sparse: if only one or two predictors matter among many, a very small \(m\) can often omit them at a node. A smaller count can be attractive when many predictors provide comparable signal or when excessive similarity among trees is a concern. These are reasons to consider a value, not rules that determine the winner. Predictive performance must be judged using an appropriate validation design; an in-sample split improvement is not a substitute for that assessment.

When comparing candidate counts, keep the interpretation clear. A change in \(m\) changes the random subset available at each split; it does not change the number \(p\) of predictors in the data. Nor does a higher probability of including a useful predictor automatically mean a proportionally lower prediction error. Trees make many dependent decisions, and forest performance reflects the combined effect of those decisions.

Common Exam Traps

• Using the wrong complement. The ratio \(\binom{p-k}{m}/\binom{p}{m}\) is the probability of selecting no useful predictors, not the probability of selecting at least one. Subtract it from 1 for the latter.
• Assuming independent inclusion. Sampling \(m\) distinct predictors without replacement means selections are dependent. Use the combination formula rather than treating each draw as an independent trial.
• Confusing \(p\) and \(m\). \(p\) is the total number of eligible predictors; \(m\) is the number considered at one node. A default rule is applied to \(p\), not to the number of useful predictors.
• Calling a common rule universally optimal. The square-root and one-third rules are starting conventions. Their use alone does not prove that a forest has the best predictive performance.
• Ignoring integer conventions. Since \(m\) must be an integer between 1 and \(p\), check how the implementation converts a fractional rule to an integer.