Why Leaf Size Matters
A regression tree makes a prediction by assigning an observation to a terminal node, also called a leaf. Under squared-error fitting, every training response in a leaf contributes to that leaf’s prediction. When a leaf contains only a few observations, one unusual response can therefore move the prediction appreciably. In a larger leaf, the same-sized change is spread across more observations.
This effect is distinct from how the tree chooses its splits. Here, take the terminal-node membership as given and examine the calculation made within each leaf. That separation makes it possible to see exactly how leaf size controls the direct sensitivity of the fitted mean.
Why the Terminal-Node Prediction Is a Mean
The prediction in a leaf is chosen to minimize the sum of squared differences between its training responses and a single constant prediction. If \(c\) is that constant, the within-leaf squared-error criterion is \(Q(c)=\sum_{i:x_i\in R_m}(y_i-c)^2\). The minimizer is the mean of the responses in the leaf.
For a fixed leaf, \(Q(c)=\sum_i(y_i-c)^2\), where the sum runs over its \(n_m\) observations.
\(\frac{dQ}{dc}=-2\sum_i(y_i-c)=-2\sum_i y_i+2n_m c\).
\(-2\sum_i y_i+2n_m c=0\), so \(c=\frac{1}{n_m}\sum_i y_i\).
\(\frac{d^2Q}{dc^2}=2n_m>0\), since a terminal node contains at least one observation. The stationary point is therefore the unique minimum.
This justification depends on the squared-error criterion and a single constant prediction within the leaf. It establishes the terminal mean; it does not say that a tree must use squared error for every task or that this mean is appropriate for every response type.
Worked Example: A Single Changed Response in a Three-Observation Leaf
A hypothetical tree has a terminal node containing responses \(40\), \(46\), and \(50\), measured in the response’s stated units. Find the fitted prediction, then calculate its change if the last response is recorded as \(62\) instead. Also compare the original prediction with the mean of the two observations left after removing the response \(50\).
Method. Use the arithmetic mean in the leaf. For a change to one response, compare the two sums while keeping the leaf membership fixed.
Original prediction. The response sum is \(40+46+50=136\), and there are \(3\) observations. Thus \(\hat{c}_m=136/3=45.333\ldots\).
Prediction after the change. Replacing \(50\) by \(62\) adds \(12\) to the response sum. The new sum is \(40+46+62=148\), so the new prediction is \(148/3=49.333\ldots\). Its change is \(49.333\ldots-45.333\ldots=4\).
Check the change directly. The response increased by \(62-50=12\), and the leaf has \(3\) observations. The mean therefore increases by \(12/3=4\), matching the calculation from the two means.
Removing the response is a different operation. With \(50\) removed, the remaining mean is \((40+46)/2=86/2=43\). Relative to the original mean, the change is \(43-136/3=(129-136)/3=-7/3=-2.333\ldots\). The leaf size as well as the response sum has changed.
Common mistake avoided: Treating a \(12\)-unit change to one response as a \(12\)-unit change to the leaf prediction. The response is one of three equally weighted values, so the leaf prediction changes by \(12/3=4\), not \(12\).
How a Single Response Affects a Leaf Mean
Suppose a fixed leaf has \(n_m\) responses and one response, \(y_j\), is changed by \(\Delta\), with no observations added or removed. The response sum increases by \(\Delta\), while the denominator \(n_m\) stays the same. Consequently, the new mean equals the old mean plus \(\Delta/n_m\).
The same calculation applies if the response moves downward: a decrease of \(\Delta\) changes the prediction by \(-\Delta/n_m\). It also gives the difference between two fitted predictions when one response is replaced by another: if \(y_j\) is replaced by \(y_j^{\mathrm{new}}\), the prediction changes by \((y_j^{\mathrm{new}}-y_j)/n_m\). The relevant quantity is the change in the response, not the response’s value alone.
Worked Example: The Same Change in Leaves of Different Sizes
Two hypothetical terminal nodes each have a fitted mean of \(50\). The first contains responses \(44\), \(50\), and \(56\). The second contains \(12\) responses: \(44\), \(46\), \(47\), \(48\), \(49\), \(50\), \(50\), \(51\), \(52\), \(53\), \(54\), and \(56\). In each node, replace a response of \(56\) by \(68\). Find the new mean in each node and compare the prediction changes.
Method. Verify each original mean, then apply the response change of \(68-56=12\) to the appropriate leaf size.
Three-observation leaf. Its original sum is \(44+50+56=150\), so its mean is \(150/3=50\). After replacing \(56\) with \(68\), the sum is \(44+50+68=162\), and the new mean is \(162/3=54\). The prediction increases by \(54-50=4\), which also equals \(12/3\).
Twelve-observation leaf. Its original sum is \(600\), as can be checked by adding the listed responses, so its mean is \(600/12=50\). The replacement raises the sum by \(12\), to \(612\). The new mean is \(612/12=51\), an increase of \(51-50=1\), which also equals \(12/12\).
Comparison and check. The same response change moves the smaller leaf’s prediction by \(4\) and the larger leaf’s prediction by \(1\). The ratio of the changes is \(4/1=4\), equal to the ratio of the leaf sizes \(12/3=4\), in the opposite direction for sensitivity.
Common mistake avoided: Comparing the response sums alone and concluding that both predictions move by \(12\). A tree predicts with the mean within each leaf, so the denominator must be included.
Leaf Size and Sampling Variability
The direct sensitivity calculation has a sampling counterpart, but that counterpart needs explicit assumptions. Suppose the leaf membership is fixed, and within that leaf the responses follow \(Y_i=\mu_i+\varepsilon_i\), where the errors \(\varepsilon_i\) are independent, have mean zero, and have common variance \(\sigma^2\). If the \(\mu_i\) values in the leaf are all the same constant \(\mu\), then the leaf mean estimates \(\mu\) without bias and has variance \(\sigma^2/n_m\).
To see the variance result, write the mean as \(\hat{c}_m=\mu+\frac{1}{n_m}\sum_i\varepsilon_i\). Independence makes the variance of the error sum equal to the sum of its variances, \(n_m\sigma^2\). Scaling that sum by \(1/n_m\) scales its variance by \(1/n_m^2\). Therefore,
Under these assumptions, doubling the leaf size halves the variance of its mean; the standard deviation is \(\sigma/\sqrt{n_m}\), so doubling the size reduces it by a factor of \(\sqrt{2}\), not by a factor of \(2\). This is a statement about variability across repeated response samples under a specified model. It is not a guarantee that a larger leaf has less prediction error in every setting: the responses may have different means within the leaf, and a fixed-partition calculation does not account for a tree selecting its leaves from the data.
Worked Example: Comparing Observed Sensitivity and Model-Based Variability
Consider two fixed hypothetical leaves. The first has responses \(82\), \(94\), \(101\), and \(103\); the second has \(16\) responses whose sum is \(1{,}600\). In each leaf, increase one response by \(8\). Separately, assume that errors within either fixed leaf are independent with mean zero and variance \(64\), and that all responses in each leaf have a common mean. Calculate both observed mean changes and the model-based standard error for each leaf mean.
Method. Use \(\Delta/n_m\) for the direct change, and use \(\sqrt{\sigma^2/n_m}\) for the standard error under the stated fixed-leaf model.
Four-observation leaf. Its response sum is \(82+94+101+103=380\), so its original mean is \(380/4=95\). Raising one response by \(8\) increases the sum to \(388\); the new mean is \(388/4=97\). The observed prediction change is \(97-95=2=8/4\). Its model-based variance is \(64/4=16\), and its standard error is \(\sqrt{16}=4\).
Sixteen-observation leaf. Its original mean is \(1{,}600/16=100\). Raising one response by \(8\) gives a new sum of \(1{,}608\) and a new mean of \(1{,}608/16=100.5\). The prediction change is \(100.5-100=0.5=8/16\). Its model-based variance is \(64/16=4\), and its standard error is \(\sqrt{4}=2\).
Check. The same response change moves the first mean by \(2\) and the second by \(0.5\). The model-based standard errors are \(4\) and \(2\), respectively. The second leaf’s standard error is half the first’s because its size is four times as large and standard error scales with the inverse square root of size.
Common mistake avoided: Confusing the observed shift from one changed response with the standard error of the mean. The shifts here are \(2\) and \(0.5\); the standard errors under the assumed model are \(4\) and \(2\). They describe different quantities.
What Small Leaves Do—and Do Not—Tell You
A small leaf makes each included response a larger share of its mean. Thus, if a response is unusually high or low, it can pull the predictions for every observation assigned to that leaf toward that response. This is why very small leaves can produce volatile terminal predictions: the fitted value can change noticeably after a modest response change, and the fixed-leaf sampling variance is larger under the equal-mean error model.
That explanation should not be extended beyond its assumptions. The variance formula \(\sigma^2/n_m\) assumes fixed membership, independent errors with common variance, and a common response mean within the leaf. If responses within the node have different means, the variance calculation alone does not describe the full estimation error around every observation’s mean. If the tree is re-grown after a response changes, its split points or terminal-node assignments may also change. The fixed-leaf sensitivity formula captures only the change in the mean when membership is held constant.
There is also a practical tradeoff. Making leaves smaller can allow a tree to represent local differences more finely, but each fitted mean then depends on fewer responses. A small leaf is not automatically wrong: its value depends on the purpose of the model, the response pattern, and the consequences of unstable predictions. The key is to recognize the cost of relying on a small number of observations.
- For a fixed leaf, a response change of \(\Delta\) changes its fitted mean by \(\Delta/n_m\).
- Removing an observation is not the same as changing its response: removal changes both the sum and the denominator.
- The fixed-leaf variance formula is a model-based result, not a complete uncertainty calculation for a tree selected from the same data.
Check Your Understanding
For each problem, treat the terminal-node membership as fixed when calculating a change to a leaf mean.
- A leaf contains responses \(12\), \(14\), \(15\), \(18\), and \(21\). Find its mean, then replace \(21\) by \(31\). What is the new mean and how much did the prediction change? (Answer: Original mean \(16\); new mean \(18\); increase \(2\).)
- A leaf has eight responses \(20\), \(22\), \(24\), \(25\), \(26\), \(27\), \(28\), and \(28\). Replace one \(28\) by \(36\). Calculate the original and new means and the change. (Answer: Original mean \(25\); new mean \(26\); increase \(1\).)
- One leaf has four responses \(6\), \(10\), \(12\), and \(12\); another has ten responses with mean \(20\). If one response in each leaf increases by \(15\), calculate the change in each leaf’s prediction. (Answer: The four-observation leaf increases by \(15/4=3.75\); the ten-observation leaf increases by \(15/10=1.5\).)
- Under fixed membership, suppose errors are independent with mean zero and variance \(36\), and the responses in each leaf have a common mean. Compare the variance and standard error of the mean for leaves of sizes \(9\) and \(25\). (Answer: Size \(9\): variance \(4\), standard error \(2\). Size \(25\): variance \(1.44\), standard error \(1.2\).)
- A leaf contains \(3\), \(5\), \(8\), \(10\), \(11\), and \(17\). Find the original mean, then remove the response \(17\) and calculate the remaining mean and its difference from the original. (Answer: Original mean \(9\); remaining mean \(37/5=7.4\); decrease \(1.6\).)