What mtry Controls
In a random forest, a tree does not necessarily examine every predictor when it chooses a split. Instead, it can draw a limited set of predictors and search for the best split among that set. The tuning quantity that sets the size of this set is called mtry. Applying the rule correctly requires keeping three ideas separate: the predictors available at a node, the split chosen at that node, and the predictors that may be available at later nodes.
This tutorial uses the standard random-forest split procedure: at each node where a split is sought, sample \(m\) distinct predictors uniformly from the \(p\) predictors, without replacement, then choose the best eligible split using the tree's splitting criterion. The random subset is drawn again at another node. Some software or specialized implementations may use different details, so a calculation should follow the stated procedure when one is given.
The candidate set contains predictors, not individual cut points. For a numeric predictor, the tree may assess multiple eligible cut points within that predictor. For a categorical predictor, it may assess eligible ways to divide its categories, subject to the tree's rules. Those possibilities do not increase mtry: if three predictors are drawn, mtry is three even if the tree evaluates many possible splits for each one.
Once the candidate set is drawn, the tree compares the allowable splits for those predictors using its splitting criterion. For example, a regression tree may seek a reduction in squared-error impurity, while a classification tree may use a classification impurity measure. The chosen predictor is the one associated with the best permitted split among the candidates. A predictor's inclusion in the candidate set does not guarantee that it will be selected.
How Candidate Sets Change Across a Tree
The mtry draw is local to a node. A predictor may be included in the candidate set at the root, be absent at a child node, and appear again at another node. The same predictor can also be chosen for splits at multiple nodes. There is no general rule that a predictor used once is removed from consideration for the rest of the tree.
For clarity, label the predictors \(X_1,\ldots,X_p\). Under uniform sampling without replacement at one node, each possible subset of \(m\) predictors is equally likely. Across nodes, the standard procedure makes fresh random draws; for probability calculations below, the draws at distinct nodes are treated as independent. The realized candidate sets and split scores determine the path of a particular tree.
Worked Example: Following Candidate Predictors Through Three Nodes
A hypothetical regression-tree node has five available predictors, \(X_1,\ldots,X_5\), and mtry is 2. At the root, the sampled set is \(\{X_2,X_5\}\). The best split using \(X_2\) reduces the node's squared-error impurity by 0.18, while the best split using \(X_5\) reduces it by 0.11. At the next node on one branch, the candidate set is \(\{X_1,X_4\}\), with reductions 0.07 and 0.12. At the next node on the other branch, the candidate set is \(\{X_2,X_3\}\), with reductions 0.09 and 0.14. Identify the selected predictor at each node.
Method. At each node, compare the scores only for predictors in that node's sampled set. The larger reduction is the better eligible split in this example.
Root. The candidates are \(X_2\) and \(X_5\). Since \(0.18 > 0.11\), the root splits on \(X_2\).
First child considered. The candidates are \(X_1\) and \(X_4\). Since \(0.12 > 0.07\), this node splits on \(X_4\).
Other child considered. The candidates are \(X_2\) and \(X_3\). Since \(0.14 > 0.09\), this node splits on \(X_3\).
Check. The selected sequence is \(X_2, X_4, X_3\), and every selected predictor was in its node's candidate set. \(X_2\) was available again at the third node, even though it had already been selected at the root. The common mistake is to assume that a predictor is removed from the tree after its first use.
Availability Probabilities
At one node, the chance that a particular predictor is among the \(m\) sampled predictors is \(m/p\). This follows by counting subsets. There are \(\binom{p}{m}\) possible candidate sets. If a particular predictor must be included, the other \(m-1\) members can be selected from the remaining \(p-1\) predictors, giving \(\binom{p-1}{m-1}\) candidate sets. Therefore,
The last equality follows by expanding the combinations: \(\binom{p-1}{m-1}/\binom{p}{m} = [(p-1)!/((m-1)!(p-m)!)]/[p!/(m!(p-m)!)] = m/p\). This is an availability probability, not the probability that the predictor is selected for the split. Selection also depends on the data at the node, the candidate predictors' possible splits, and the splitting criterion.
If the same predictor is considered across \(k\) independently sampled node sets, the probability it is absent from one node is \(1-m/p\). It is absent from all \(k\) sets with probability \((1-m/p)^k\). Taking the complement gives the probability it is available at least once:
This formula concerns specified node draws, such as the draws at \(k\) nodes along a hypothetical path. It does not say that a predictor will be available at \(k\) actual nodes of a tree: a tree's structure and which nodes exist depend on the splits made earlier.
Worked Example: Availability Along a Path
A hypothetical forest model has \(p=8\) predictors and mtry \(m=3\). Consider a specified sequence of four nodes at which a split is sought. Assume the candidate sets are drawn independently at those nodes. For predictor \(X_7\), calculate (a) the chance it is available at any one of the nodes, (b) the chance it is never available in the four draws, and (c) the chance it is available at least once.
(a) One node. The inclusion probability is \(m/p=3/8=0.375\).
(b) No availability in four draws. At one node, the probability \(X_7\) is absent is \(1-3/8=5/8\). By independence, the probability it is absent from all four sets is \( (5/8)^4=625/4096\), approximately \(0.1526\).
(c) Availability at least once. Use the complement: \(1-625/4096=3471/4096\), approximately \(0.8474\).
Check. The two four-node outcomes, never available and available at least once, are complementary, and their probabilities sum to \(1\). The common mistake is to multiply \(3/8\) by four and call the result the probability of availability at least once. That gives an expected number of available nodes, not the probability of at least one occurrence; the complement calculation handles repeated availability correctly.
Choosing a Split Is Not the Same as Drawing Predictors
The candidate draw and the split search are two distinct steps. First the randomization determines which predictors can compete at the node. Then the tree evaluates eligible splits among those predictors and selects the best one under its criterion. If the strongest predictor overall is not drawn, it cannot be selected at that node, even if it would have produced a better split than every available predictor.
The following procedure is useful when a question gives a series of candidate sets or split scores:
Use the stated mtry draw for that node. Do not carry a prior node's set forward unless the problem explicitly says to do so.
Use the stated impurity measure or split score, and ignore predictors outside the candidate set at that node.
For the next node, use its own candidate draw. A predictor may reappear or be absent.
Worked Example: Candidate Sets Versus Selected Predictors
A hypothetical classification tree has four predictors, \(A,B,C,D\), and mtry is 2. At three successive nodes where splits are sought, the candidate sets are \(\{A,B\}\), \(\{A,C\}\), and \(\{B,D\}\), respectively. The best available splits have impurity reductions as follows: at the first node, \(A: 0.20\), \(B: 0.16\); at the second, \(A: 0.04\), \(C: 0.13\); at the third, \(B: 0.09\), \(D: 0.06\). Determine the selected predictor at each node and whether \(A\) is selected whenever it is available.
First node. Compare \(0.20\) for \(A\) with \(0.16\) for \(B\). Since \(0.20>0.16\), select \(A\).
Second node. Compare \(0.04\) for \(A\) with \(0.13\) for \(C\). Since \(0.13>0.04\), select \(C\), not \(A\).
Third node. Compare \(0.09\) for \(B\) with \(0.06\) for \(D\). Since \(0.09>0.06\), select \(B\).
Check. The selected sequence is \(A,C,B\). Predictor \(A\) is available at the first two nodes, but it is selected at only the first. Availability permits a predictor to compete; it does not predetermine the winner. The common mistake is to count every appearance of \(A\) in a candidate set as a split made on \(A\).
Boundary Cases and Exam Traps
At the two ends of the allowed range, the rule is especially easy to check. If \(m=p\), every predictor is available at every node where a split is sought, so feature subsampling does not restrict the split search. If \(m=1\), exactly one predictor is available at each such node; the tree can still assess eligible split points for that predictor, but cannot compare it with a different predictor at the same node.
A useful count is the number of possible candidate sets, \(\binom{p}{m}\), when each set of \(m\) distinct predictors is equally likely. For example, if \(p=5\) and \(m=2\), there are \(\binom{5}{2}=10\) possible sets. This is not the number of possible splits in the tree: the number of cut points or partitions within each predictor is a separate matter.
- Wrong: “With \(m=2\), the tree tests two cut points.” Correction: It samples two predictors; it may test multiple cut points for each.
- Wrong: “The same two predictors are used at every node.” Correction: Under the standard rule, a fresh candidate set is sampled at each node where a split is sought.
- Wrong: “A predictor included in the draw must be selected.” Correction: It is only eligible; the best available split determines the chosen predictor.
- Wrong: “The chance of being selected is \(m/p\).” Correction: \(m/p\) is the chance of being available at one node. Selection depends on split quality and cannot generally be calculated from \(m\) and \(p\) alone.
For an actuarial modeling problem, this distinction matters when interpreting how a tree was constructed or checking a claimed split sequence. The mtry rule limits which variables can compete locally; it does not impose a fixed set of predictors on the whole tree. These mechanics are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, distinguish predictor availability from the split ultimately chosen.
- A forest has \(p=10\) predictors and \(m=4\). What is the chance that a specified predictor is available at one node? (Answer: \(0.4000\))
- A model has \(p=6\) predictors and \(m=2\). At two independently sampled nodes, what is the probability a specified predictor is absent at both, and what is the probability it appears at least once? (Answer: \(4/9\) absent at both; \(5/9\) available at least once.)
- There are \(p=7\) predictors and \(m=3\). How many candidate sets are possible, and what is the probability of one specified set? What is the probability that a specified predictor is in the set? (Answer: 35 sets; \(1/35\) for a specified set; \(3/7\) for a specified predictor to be included.)
- For \(p=5\), \(m=2\), and three independent node draws, calculate the probability a specified predictor is available exactly twice and at least once. (Answer: \(0.2880\) exactly twice; \(0.7840\) at least once.)
- A regression tree has \(p=4\), \(m=2\). At one node the candidate set is \(\{A,C\}\), with best split reductions \(A:0.07\), \(C:0.12\). At the next node the set is \(\{A,D\}\), with reductions \(A:0.15\), \(D:0.10\). Which predictors are selected, and does \(A\)'s first-node availability require it to be selected there? (Answer: \(C\) at the first node and \(A\) at the second; no, availability does not require selection.)