Why Leaf Size and Bootstrap Size Matter
A forest combines predictions from many trees, but the trees’ behavior depends on how each tree is grown and on the data available to it. The minimum terminal-node size controls how finely an individual tree can partition the predictor space. The bootstrap sample size controls how many draws, including repeated observations, are used to grow each tree. Both choices can affect prediction smoothness, tree diversity, and the variance of the averaged prediction.
These effects involve a balance. Smaller leaves can represent local patterns more closely, but their predictions may be sensitive to individual observations. Larger leaves tend to smooth predictions, but may obscure real structure. A smaller bootstrap sample can make trees less alike, but may also leave each tree with less information. Neither setting has a universally best value.
Tree-growing conventions matter. A setting called “minimum leaf size” usually requires every resulting terminal node to contain at least \(q\) observations. Some software instead applies a related rule to the number of observations required to attempt a split, or counts duplicated bootstrap rows as separate observations. Always interpret the stated rule. In the examples below, \(q\) counts the bootstrap observations in a node, including multiplicities.
How Leaf Size Changes Individual Trees
In a regression tree, the prediction in a terminal node is typically the mean response among the training observations in that node. A split creates smaller regions with separate means. Requiring more observations in every terminal node rules out some splits, so the tree usually has fewer, larger regions and a less locally variable prediction. Requiring fewer observations permits finer partitions, which can capture local changes but also makes node estimates more dependent on the particular sample.
In a classification tree, a terminal node’s class proportions can be used as estimated class probabilities; the predicted class is often the class with the largest proportion. Small leaves can produce more extreme proportions, including zero or one, while larger leaves generally produce more stable proportions. A large leaf can, however, combine cases from different underlying groups.
Worked Example: A Regression Split and the Leaf-Size Rule
A regression-tree node contains four observations ordered by a predictor. Their responses are \(2, 4, 6,\) and \(8\). A candidate split after the second observation would create two child nodes of size 2. Compare the result under minimum terminal-node sizes \(q=2\) and \(q=3\). Assume this is the candidate split being evaluated and that the node is not subject to another stopping rule.
Method. Check whether each child meets the minimum size, then calculate the mean response in each resulting leaf. This isolates the split’s effect; it does not assume that every eligible split must be selected.
For \(q=2\), both child nodes meet the requirement because each has 2 observations. The left-node prediction is \[ \frac{2+4}{2}=\frac{6}{2}=3, \] and the right-node prediction is \[ \frac{6+8}{2}=\frac{14}{2}=7. \]
For \(q=3\), each child would need at least 3 observations, but each has only 2. The proposed split is therefore prohibited. If this node remains terminal, its prediction is the mean of all four responses: \[ \frac{2+4+6+8}{4}=\frac{20}{4}=5. \]
The calculation checks: the two child means average to the parent mean, \((3+7)/2=5\), because the child nodes have equal sizes. The smaller minimum size permits a more detailed prediction, 3 or 7 depending on the side of the split; the larger size forces a single, smoother prediction of 5. Common mistake avoided: treating \(q\) as a required number of observations in the parent node rather than in each resulting terminal node.
Worked Example: Leaf Size and Classification Probabilities
A classification-tree node contains 8 cases: 4 belong to class A and 4 to class B. A candidate split produces two child nodes, each with 4 cases. The first child contains 3 A cases and 1 B case; the second contains 1 A case and 3 B cases. Compare the split when \(q=4\) and \(q=5\), using Gini impurity \(G=1-\sum_k p_k^2\), where \(p_k\) is the proportion in class \(k\).
Method. First check whether the children meet the size constraint, then calculate the parent and child impurities. A weighted average of child impurities uses each child’s proportion of the parent’s cases.
At the parent, \(p_A=p_B=4/8=0.5\), so \[ G_{\text{parent}}=1-(0.5^2+0.5^2)=1-0.5=0.5. \]
For \(q=4\), both children are eligible. The first child has class proportions \(3/4\) and \(1/4\), so its impurity is \[ 1-\left((3/4)^2+(1/4)^2\right)=1-(9/16+1/16)=6/16=0.375. \] The second child has proportions \(1/4\) and \(3/4\), giving the same impurity, \(0.375\). Since each child contains half the cases, the weighted child impurity is \[ (4/8)(0.375)+(4/8)(0.375)=0.375. \] The impurity reduction is \(0.5-0.375=0.125\).
For \(q=5\), neither child meets the minimum size of 5. The split is prohibited, so the node remains unsplit under this candidate. Its class-A probability is \(4/8=0.5\), rather than the \(3/4=0.75\) probability in the first child when the \(q=4\) split is allowed. Common mistake avoided: concluding that a split is allowed because the parent has at least \(q\) cases; the rule must be checked for each resulting child.
How Bootstrap Sample Size Changes the Cases Available
A bootstrap sample is formed by drawing training cases with replacement. It is important to distinguish the number of draws, \(B\), from the number of distinct training cases represented. For any one case, the probability of not being drawn on a particular draw is \(1-1/n\). Therefore, the probability it is absent from all \(B\) draws is \((1-1/n)^B\), and the probability it appears at least once is \(1-(1-1/n)^B\).
This expectation follows by defining an indicator equal to 1 if a particular case appears and 0 otherwise. Its expected value is its probability of appearing. Summing the \(n\) indicators gives the number of distinct cases, and linearity of expectation gives the displayed formula; independence among the indicators is not required.
Worked Example: Comparing Two Bootstrap Sample Sizes
A training set has \(n=6\) cases. Compare bootstrap samples of \(B=3\) and \(B=6\) draws. Calculate the expected number of distinct cases and the probability that a specified case is omitted.
Method. Apply the inclusion probability \(1-(1-1/n)^B\), multiply by \(n\) for the expected number of distinct cases, and use the complementary probability for omission.
For \(B=3\), the probability a specified case appears is \[ 1-(5/6)^3=1-125/216=91/216\approx 0.4213. \] The expected number of distinct cases is \[ 6(91/216)=546/216=91/36\approx 2.5278. \] The probability the specified case is omitted is \(125/216\approx 0.5787\). Thus the expected number of cases omitted is \(6(125/216)=125/36\approx 3.4722\).
For \(B=6\), the probability a specified case appears is \[ 1-(5/6)^6=1-15625/46656=31031/46656\approx 0.6651. \] The expected number of distinct cases is \[ 6(31031/46656)=31031/7776\approx 3.9906. \] The omission probability is \(15625/46656\approx 0.3349\), giving an expected \(6(15625/46656)=15625/7776\approx 2.0094\) omitted cases.
The results check: the expected number of distinct cases plus the expected number omitted equals 6 in both settings. Although \(B=6\) draws are made in the second sample, the expected number of distinct cases is only about 3.9906. Common mistake avoided: treating bootstrap draws as distinct observations.
As \(B\) increases, each tree generally has access to more distinct training cases, though the expected number cannot exceed \(n\). A smaller \(B\) can make trees more different because each is grown from a more limited sample; it can also make individual trees less stable or less accurate. A larger \(B\) may produce stronger trees, but trees can become more alike because their samples overlap more. These are tendencies, not guarantees: the predictors, response pattern, tree-growing rules, and random feature selection also affect tree similarity.
The minimum leaf size interacts with bootstrap size. If a bootstrap sample contains few distinct cases, a large \(q\) may prevent many candidate splits. Conversely, a small \(q\) can allow highly detailed trees, but the amount of distinct information available still depends on the bootstrap sample. The total number of draws alone does not describe the sample’s distinct information.
Connecting Tree Choices to Forest Prediction Variance
A forest averages predictions from \(M\) trees. If the tree predictions at a fixed input \(x\) have equal variance \(\sigma_T^2\) and every pair has the same correlation \(\rho\), the variance of their average is \[ \operatorname{Var}\left(\frac{1}{M}\sum_{b=1}^{M}T_b(x)\right) =\sigma_T^2\left[\rho+\frac{1-\rho}{M}\right]. \] Here \(T_b(x)\) is the prediction from tree \(b\). This formula assumes equal variances and a common pairwise correlation at the input under consideration; real forests need not meet those assumptions exactly.
Derivation of the Equal-Correlation Formula
The variance of a sum is the sum of all variances and covariances. There are \(M\) individual tree variances, each \(\sigma_T^2\), and \(M(M-1)\) ordered off-diagonal covariance terms, each \(\rho\sigma_T^2\). Dividing by \(M^2\) for the average gives
The result separates two influences. Increasing \(M\) reduces the contribution \((1-\rho)\sigma_T^2/M\), but the term \(\rho\sigma_T^2\) remains. Thus, changing the number of trees alone cannot remove the variance contribution associated with correlated tree predictions. Leaf size and bootstrap size can affect both the variance of an individual tree and the correlation among trees, so their net effect on forest variance is not determined by either effect alone.
Worked Example: Comparing Two Illustrative Forest Settings
At a fixed prediction point, suppose two hypothetical forest settings each use \(M=25\) trees. Under a smaller-leaf setting, assume the tree prediction variance is \(\sigma_T^2=16\) and the pairwise correlation is \(\rho=0.40\). Under a larger-leaf setting, assume \(\sigma_T^2=9\) and \(\rho=0.70\). Calculate and compare the forest prediction variances under the equal-correlation model.
Method. Substitute each setting’s variance and correlation into the derived formula. These stipulated values illustrate the calculation; they are not universal consequences of either leaf-size choice.
For the smaller-leaf setting, \[ 16\left[0.40+\frac{1-0.40}{25}\right] =16(0.40+0.024) =16(0.424) =6.784. \]
For the larger-leaf setting, \[ 9\left[0.70+\frac{1-0.70}{25}\right] =9(0.70+0.012) =9(0.712) =6.408. \]
Under these assumptions, the larger-leaf setting has the lower forest prediction variance, \(6.408\) versus \(6.784\), even though its tree correlation is higher. The lower individual-tree variance more than offsets the higher correlation in this comparison. The arithmetic checks directly against the formula. Common mistake avoided: deciding which forest has lower variance by comparing correlation alone.
Practical Interpretation and Exam Traps
The choice of \(q\) and \(B\) should be interpreted as a joint modeling decision. Smaller leaves can reduce smoothing bias when the response changes locally, while increasing sensitivity to the particular training sample. Larger leaves can stabilize predictions while missing local structure. A smaller bootstrap sample can increase tree diversity while weakening individual trees; a larger one may strengthen trees while making their predictions more correlated. Because these effects can offset one another, there is no general rule that “smaller is better” or “larger is better.”
A common arithmetic trap is to assume that a bootstrap sample with \(B\) draws contains exactly \(B\) distinct cases. For example, with \(n=100\) and \(B=50\), that incorrect calculation would claim 50 distinct cases. The correct expected count is \[ 100[1-(0.99)^{50}]\approx 100(1-0.6050)\approx 39.50. \] The probability a specified case is omitted is \((0.99)^{50}\approx 0.6050\). The calculation concerns an expectation, not a guarantee that every sample contains exactly 39.50 distinct cases.
When interpreting a comparison, check the actual rule and the sample unit being counted. Confirm whether \(q\) applies to each child node, and whether the implementation counts repeated bootstrap draws as separate rows. Then distinguish a tree-level effect from a forest-level effect: a change in tree variance does not by itself determine forest variance unless the change in tree correlation is also considered.
Check Your Understanding
Work each problem by checking the stated size rule before calculating predictions or probabilities.
- A regression node has 12 observations. A proposed split creates children of sizes 4 and 8, with response sums 20 and 48. Is the split allowed for \(q=5\) and for \(q=4\)? If it is allowed, calculate both child means. (Answer: For \(q=5\), no; for \(q=4\), yes, with child means 5 and 6.)
- A classification parent has 10 cases, 6 in class A and 4 in class B. A proposed split creates two children of size 5: the first has 4 A and 1 B, and the second has 2 A and 3 B. Calculate the parent Gini impurity and the weighted child impurity. Is the split size-eligible for \(q=5\) and \(q=6\)? (Answer: Parent Gini \(=0.48\); weighted child Gini \(=0.40\); impurity reduction \(=0.08\); eligible for \(q=5\), not \(q=6\).)
- A training set has \(n=10\) cases, and each tree uses \(B=5\) bootstrap draws. Find the expected number of distinct cases and the probability that a specified case is omitted. (Answer: \(10[1-(0.9)^5]=4.0951\) expected distinct cases; omission probability \(0.9^5=0.5905\), to four decimals.)
- At a fixed input, a forest has \(M=16\) trees with individual prediction variance \(\sigma_T^2=25\) and common pairwise correlation \(\rho=0.20\). Find the forest prediction variance under the equal-correlation model. (Answer: \(25[0.20+0.80/16]=6.25\).)
- Two hypothetical settings each use \(M=10\) trees. Setting A has \(\sigma_T^2=12\) and \(\rho=0.25\); setting B has \(\sigma_T^2=8\) and \(\rho=0.50\). Calculate each forest prediction variance and identify the lower one. (Answer: A: \(12[0.25+0.75/10]=3.9\); B: \(8[0.50+0.50/10]=4.4\); setting A is lower.)