Why Randomize the Predictors at a Split?
A tree chooses a split from the predictors available to it. If one predictor repeatedly offers the strongest split, many trees in a forest may rely on that same predictor near the top of the tree. Their predictions can then respond similarly to changes in the data. Random forests introduce another source of variation: at each split, only a randomly selected subset of the predictors is allowed to compete. The treatment here is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
This randomization does not change the split criterion. A classification tree still compares candidate splits using its chosen measure of class separation, and a regression tree still compares them using its chosen measure of response variation. The change is that the tree searches only among the predictors in the current candidate subset. If a strong predictor is not in that subset, it cannot be chosen at that node.
The candidate set is selected separately from the split calculation. Drawing a predictor into the candidate set makes it eligible; it does not guarantee that it supplies the chosen split. A predictor may be available but lose to another candidate, have no admissible split, or tie with another candidate. Conversely, a predictor that is not in the set cannot be selected at that node, regardless of how strong it might have been.
This mechanism adds feature-selection randomness to the randomness already introduced by fitting trees to different samples, such as bootstrap samples. It can be especially relevant when a small number of predictors are consistently strong: without feature subsampling, those predictors may repeatedly dominate the competition. Limiting the competition gives other predictors opportunities to be selected when the strongest options are absent.
Probability of Being a Candidate
Because \(m\) distinct predictors are sampled uniformly from \(p\), each predictor has the same chance of appearing in the candidate set. For a specified predictor, the probability of inclusion is \(m/p\). The probability of exclusion is \(1-m/p\). These are probabilities of availability, not probabilities of winning the split competition.
For a group of \(q\) predictors, the number of candidates from that group follows a hypergeometric distribution: the \(m\) candidates are drawn without replacement from a collection containing \(q\) members of the group and \(p-q\) other predictors. In particular, the chance that all \(m\) candidates come from outside the group is the number of possible all-outside subsets divided by the number of possible subsets overall.
where the numerator is zero if \(m>p-q\). Thus, the chance that at least one member of the group is available is one minus this exclusion probability. The expected number of candidates from the group is \(mq/p\): each of its \(q\) predictors has inclusion probability \(m/p\), so adding their indicator variables gives that expectation.
Why the Inclusion Formula Holds
There are \(\binom{p}{m}\) equally likely candidate sets. To count the sets containing one specified predictor, include that predictor and choose the remaining \(m-1\) candidates from the other \(p-1\) predictors. Dividing this count by the total number of sets gives the inclusion probability.
The derivation depends on uniform sampling of distinct predictors. If candidates were drawn independently with replacement, the probability would be different because the same predictor could appear more than once in a draw. Standard node-level feature subsampling uses a set of distinct candidate predictors, not repeated copies of one predictor.
Worked Example: A Dominant Predictor Is Not Always Available
At a node in a hypothetical claim-severity regression tree, there are \(p=8\) predictors. The split gains, if the predictors are considered, are fixed for this calculation and rank as follows: \(X_1\) has gain 18, \(X_2\) has gain 12, \(X_3\) has gain 9, \(X_4\) has gain 6, \(X_5\) has gain 4, \(X_6\) has gain 3, \(X_7\) has gain 2, and \(X_8\) has gain 1. All are assumed to have an admissible split. The node draws \(m=3\) candidates without replacement. Find the chance that \(X_1\) is available, and the expected gain of the best available split.
Method. First calculate availability using uniform subset sampling. Then group candidate sets according to the highest-ranked predictor included. This is not an average over response outcomes; it is an average over the random candidate sets, holding the stated gains fixed.
There are \(\binom{8}{3}=56\) possible candidate sets. The number containing \(X_1\) is \(\binom{7}{2}=21\), since the other two candidates are chosen from the seven remaining predictors. Therefore,
For the best available gain to come from \(X_1\), the set must include \(X_1\), giving 21 sets. For the best available gain to come from \(X_2\), \(X_1\) must be absent and \(X_2\) present; the other two candidates must be chosen from \(X_3,\ldots,X_8\), giving \(\binom{6}{2}=15\) sets. Continuing in rank order gives 10 sets for \(X_3\), 6 for \(X_4\), 3 for \(X_5\), and 1 for \(X_6\). If the set contains only \(X_7\) and \(X_8\) as its highest-ranked members, it must also contain another predictor above them, so they cannot be the best available predictor in a three-predictor set. The counts sum to \(21+15+10+6+3+1=56\), as required.
The expected best available gain is therefore
The final gain is rounded to two decimal places. With all predictors available, the best gain would be 18; with only three candidates, the expected best available gain is 12.48 under the stated fixed-gain assumptions. The common mistake this avoids is treating \(X_1\)'s availability as certain just because it is the strongest predictor. Availability is random, and even when available, a predictor's realized split still depends on the competition.
How Feature Subsampling Can Spread Split Opportunities
A useful way to assess the mechanism is to define a group of dominant predictors in advance and calculate how often the group is absent. When the whole group is absent, the tree must choose among other available predictors if a split is made. When one or more dominant predictors are present, they can still win. The calculation quantifies opportunity; it does not assert that the alternatives are equally predictive or that the resulting split will be useful.
For two specified predictors, the probability that both appear in the same candidate set is \(m(m-1)/(p(p-1))\). This follows by treating the first predictor's inclusion chance as \(m/p\), then noting that, conditional on its inclusion, \(m-1\) of the remaining \(p-1\) slots are filled by the other predictor with probability \((m-1)/(p-1)\). This joint probability helps explain why reducing the candidate-set size makes it less common for a pair of predictors to compete directly.
Worked Example: How Often Are Dominant Predictors Omitted?
A hypothetical underwriting tree has \(p=6\) predictors. Two predictors, \(X_1\) and \(X_2\), have been identified as dominant at this particular node. Compare candidate sets of size \(m=2\) and \(m=4\). For each size, calculate the chance that neither dominant predictor is available, the chance that both are available, and the expected number of dominant predictors in the set.
Method. Count subsets without replacement. There are \(\binom{6}{2}=15\) sets when \(m=2\). The number containing neither dominant predictor is \(\binom{4}{2}=6\), and the number containing both is \(\binom{4}{0}=1\). The remaining \(15-6-1=8\) contain exactly one. Thus, for \(m=2\), the probabilities of neither, exactly one, and both are \(6/15\), \(8/15\), and \(1/15\), respectively. The probability neither is available is \(0.4000\); the probability both are available is \(0.0667\), with probabilities rounded to four decimal places.
The expected number of dominant predictors is \(mq/p=2(2)/6=2/3\), or \(0.6667\) to four decimal places. This also checks against the subset counts: \(0(6/15)+1(8/15)+2(1/15)=10/15=2/3\).
For \(m=4\), there are \(\binom{6}{4}=15\) sets. Neither dominant predictor is available only if all four non-dominant predictors are selected, so there is \(\binom{4}{4}=1\) such set. Both dominant predictors are available if the other two candidates are chosen from the four non-dominant predictors, giving \(\binom{4}{2}=6\) sets. The remaining \(15-1-6=8\) sets contain exactly one dominant predictor. The probabilities of neither, exactly one, and both are therefore \(1/15\), \(8/15\), and \(6/15\). The probability neither is available is \(0.0667\), and the probability both are available is \(0.4000\).
The expected number of dominant predictors is \(4(2)/6=4/3=1.3333\) to four decimal places; the subset-count check is \(0(1/15)+1(8/15)+2(6/15)=20/15=4/3\). With \(m=2\), the dominant group is absent more often and the two dominant predictors are rarely together in the candidate set. With \(m=4\), the group is almost always represented and the pair competes together more often. This comparison describes candidate-set behavior, not a recommendation about which value to use.
The point is not that every omitted dominant predictor is replaced by a good alternative. If the remaining predictors contain little useful information, the available split may be weak. But when other predictors carry overlapping or complementary signal, random omission can let them influence tree structure. This may make fitted trees less alike, particularly when their top splits would otherwise repeatedly depend on the same few predictors.
Correlated predictors illustrate the idea. If two predictors carry similar information, they may be plausible substitutes at a node. A candidate set containing only one gives that predictor the opportunity to represent their shared signal. Different nodes or trees can expose different members of the pair, so the forest's split usage may be spread across them. This does not mean the predictors are interchangeable in every part of the feature space, nor does it show that either predictor has a causal effect.
Worked Example: Correlated Proxies and Split Competition
At a hypothetical classification-tree node, \(p=5\) predictors are available. Two correlated predictors, \(X_1\) and \(X_2\), have equal split scores, each higher than the scores for \(X_3,X_4,X_5\). The node samples \(m=2\) predictors. Assume that if both \(X_1\) and \(X_2\) are candidates, a tie is broken by a fair random choice; if just one is present with a lower-scoring predictor, the correlated predictor wins. Find the probabilities that \(X_1\) and \(X_2\) supply the selected split, and compare them with the chance that either correlated predictor supplies it.
Method. Enumerate the \(\binom{5}{2}=10\) equally likely candidate pairs. One pair contains both correlated predictors: \(\{X_1,X_2\}\). Three pairs contain \(X_1\) and one of \(X_3,X_4,X_5\); three contain \(X_2\) and one of those predictors; the remaining three pairs contain only \(X_3,X_4,X_5\).
If \(X_1\) is paired with a lower-scoring predictor, it wins in three of the ten sets. If the set is \(\{X_1,X_2\}\), it wins the tie with probability \(1/2\), contributing \(1/20\). Therefore, \(P(X_1\text{ supplies the split})=3/10+1/20=7/20=0.3500\). By symmetry, \(P(X_2\text{ supplies the split})=0.3500\).
The two events are mutually exclusive because a single split is selected. Thus, the probability either correlated predictor supplies the split is \(0.3500+0.3500=0.7000\). The other \(0.3000\) of candidate sets contain only the lower-scoring predictors, and one of those supplies the split. As a check, the probabilities for the possible outcomes sum to \(0.3500+0.3500+0.3000=1.0000\). The common mistake this avoids is assuming that each correlated predictor must always be selected merely because it has a high score; candidate availability and the tie rule both matter.
What Random Feature Selection Does—and Does Not—Guarantee
Feature subsampling can reduce repeated reliance on dominant predictors because it sometimes removes them from a node's competition. It may also reduce the frequency with which correlated predictors appear together. These mechanisms can create more variation in tree structures and predictions. Since ensemble behavior depends in part on how similar the trees are, this added variation can help reduce their dependence.
It is not a guarantee that tree correlation will fall by a particular amount, or that prediction error will improve. Trees still share the training data and may find similar splits among the predictors that remain available. A dominant predictor may be included often enough to keep winning. A weaker candidate set can also produce less accurate individual trees. The net effect depends on the signal, predictor relationships, split rules, and other modeling choices.
Common Exam Trap: Sampling With Replacement
A frequent error is to treat the \(m\) candidates as \(m\) independent draws with replacement. For example, with \(p=8\) and \(m=3\), that incorrect model gives the inclusion probability for one predictor as
This calculation is wrong for a candidate set sampled without replacement: it counts the chance that a predictor appears at least once in repeated independent draws, where duplicates are possible. The correct calculation is \(m/p=3/8=0.375\). Another related error is to equate inclusion probability with split-selection probability. The former only says that a predictor is allowed to compete; its score and the other candidates determine whether it wins.
For node-level feature subsampling, treat the candidates as a uniformly chosen set of distinct predictors.
Use \(m/p\) for one specified predictor, or a combination count for a group of predictors.
Availability determines which predictors can compete; split scores determine which candidate wins.
Random omission can diversify trees, but it does not by itself prove lower prediction error or lower tree correlation.
Check Your Understanding
For each problem, treat candidates as a uniformly sampled set of distinct predictors. Probabilities are rounded to four decimal places where needed.
- A node has \(p=10\) predictors and samples \(m=4\). What is the probability that a specified predictor is available? (Answer: \(4/10=0.4000\))
- A forest has \(p=7\) predictors, of which two are considered dominant at a node. If \(m=3\), what is the probability that neither dominant predictor is available? (Answer: \(\binom{5}{3}/\binom{7}{3}=10/35=0.2857\))
- There are \(p=8\) predictors, and a node samples \(m=2\). Find the probability that two specified predictors are both available, and the probability that at least one of them is available. (Answer: both \(=\binom{6}{0}/\binom{8}{2}=1/28=0.0357\); at least one \(=1-\binom{6}{2}/\binom{8}{2}=13/28=0.4643\))
- Of \(p=9\) predictors, three belong to a dominant group. A node samples \(m=3\). What is the probability that at least one group member is available? (Answer: \(1-\binom{6}{3}/\binom{9}{3}=64/84=16/21=0.7619\))
- A node has \(p=12\) predictors and samples \(m=3\). Four predictors belong to a specified group. What is the expected number of candidates from that group? (Answer: \(mq/p=3(4)/12=1\))