Regression-Tree Predictions Within A Leaf

Why a Regression Tree Predicts One Value Within a Leaf

A regression tree assigns observations to terminal leaves and gives a prediction for each leaf. Once the leaf membership is fixed, the prediction problem is simple: choose one number to represent the response values of the training observations in that leaf. Under squared-error loss, the number that minimizes the total training error is the arithmetic mean of those responses.

This result explains the prediction attached to a leaf; it does not determine which splits create the leaf. The discussion here treats the leaf’s membership as given and focuses on choosing its constant prediction. In practice, a new observation routed to that leaf receives the same leaf prediction as every other observation routed there.

Definition: For a terminal leaf containing $n$ training observations with responses $y_1,\ldots,y_n$, a constant leaf prediction is a number $c$ used as the prediction for each of those observations. Under unweighted squared-error loss, the leaf’s residual sum of squares at prediction $c$ is $\sum_{i=1}^{n}(y_i-c)^2$. The squared-error-optimal leaf prediction is the value of $c$ that minimizes this sum.

The $y_i$ values are the observed responses, and $c$ is a proposed prediction on the same response scale. For example, if the response is annual cost measured in dollars, both each $y_i$ and the resulting leaf prediction are measured in dollars. The objective here weights each observation equally. If observations have different weights, or if the modelling objective uses a different loss function, the result need not be the ordinary arithmetic mean.

Why the Mean Minimizes Squared Error

Let $\bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i$ be the mean response in a fixed leaf. For any proposed constant $c$, write each deviation from $c$ as $(y_i-c)=(y_i-\bar{y})+(\bar{y}-c)$, then expand the sum of squares.

\begin{aligned} \sum_{i=1}^{n}(y_i-c)^2 &=\sum_{i=1}^{n}\bigl((y_i-\bar{y})+(\bar{y}-c)\bigr)^2\\ &=\sum_{i=1}^{n}(y_i-\bar{y})^2 +2(\bar{y}-c)\sum_{i=1}^{n}(y_i-\bar{y}) +n(\bar{y}-c)^2. \end{aligned}

By the definition of $\bar{y}$, the deviations from the mean sum to zero: $\sum_{i=1}^{n}(y_i-\bar{y})=\sum_{i=1}^{n}y_i-n\bar{y}=0$. Therefore, the middle term vanishes, leaving an identity that compares the squared error at any $c$ with the squared error at the mean.

\sum_{i=1}^{n}(y_i-c)^2 = \sum_{i=1}^{n}(y_i-\bar{y})^2+n(c-\bar{y})^2 \geq \sum_{i=1}^{n}(y_i-\bar{y})^2.

Since $n(c-\bar{y})^2$ is nonnegative and is zero only when $c=\bar{y}$, the mean is the unique minimizer when $n>0$. This identity also gives a useful check: using a different constant increases the sum of squared errors by exactly $n(c-\bar{y})^2$. It is not necessary to calculate every residual again to compare that alternative with the mean.

Theorem: For a nonempty, fixed leaf with equally weighted training responses and a constant prediction, the arithmetic mean of the responses uniquely minimizes the sum of squared errors. This result applies to the stated objective; it does not say that the mean minimizes absolute error, nor that a leaf mean is guaranteed to be the best prediction on future data.

Calculating and Checking Leaf Predictions

To calculate a leaf prediction, use only the responses of the training observations assigned to that leaf. Add those responses and divide by their count. Then, if a check is needed, calculate the residuals from the proposed prediction and square them. Keep the response scale and units clear throughout: a mean of responses recorded in thousands of dollars is a prediction in thousands of dollars, not dollars.

Identify the leaf’s training responses.
Use the observations assigned to this leaf, not the responses from other leaves or from the entire data set.

Calculate the response mean.
Add the $n$ responses and divide by $n$. This is the squared-error-optimal constant for the leaf.

Interpret the result.
State that observations routed to this leaf receive the calculated prediction, on the response’s original recorded scale.

Check when useful.
Compute the residuals and their squared sum, or compare a proposed alternative using the mean-deviation identity.

Worked Example: Finding and Checking a Leaf Prediction

A hypothetical tree has a terminal leaf containing four training observations with annual repair costs of $180, $240, $300, and $360. Find the leaf prediction and compare its squared error with the squared error from predicting $250 for all four observations. Use dollars throughout.

Method. Because the leaf is fixed, the observations have equal weight, and the requested criterion is squared error, calculate the mean and use it as the optimal constant. The common mistake to avoid is selecting a convenient round number without checking its total squared error.

Step 1: Calculate the mean. The response total is $180+240+300+360=1{,}080$ dollars, and there are four observations. Thus the mean is $1{,}080/4=270$ dollars. The leaf prediction is $270.

Step 2: Calculate squared error at the mean. The residuals are $180-270=-90$, $240-270=-30$, $300-270=30$, and $360-270=90$ dollars. Their squared errors are $8{,}100$, $900$, $900$, and $8{,}100$ dollars squared. The total is $8{,}100+900+900+8{,}100=18{,}000$ dollars squared.

Step 3: Calculate squared error at $250. The residuals are $-70$, $-10$, $50$, and $110$ dollars. The squared errors are $4{,}900$, $100$, $2{,}500$, and $12{,}100$ dollars squared. Their total is $4{,}900+100+2{,}500+12{,}100=19{,}600$ dollars squared.

Check. The alternative prediction is $20 below the mean, so the identity gives an increase of $n(c-\bar{y})^2=4(250-270)^2=4(400)=1{,}600$ dollars squared. Indeed, $18{,}000+1{,}600=19{,}600$. The mean’s squared error is smaller, as required.

Worked Example: Calculating Predictions for Two Existing Leaves

A hypothetical tree already has two terminal leaves. The training responses, measured in thousands of dollars, are 12, 18, and 21 in Leaf A, and 29 and 40 in Leaf B. Find the prediction for each leaf. Then determine the prediction for a new observation that the tree routes to Leaf B.

Method. Calculate a separate mean using each leaf’s own training responses. The common mistake to avoid is pooling all five responses and assigning the resulting overall mean to both leaves.

Step 1: Calculate Leaf A’s prediction. Its response total is $12+18+21=51$ thousand dollars. There are three responses, so its mean is $51/3=17$ thousand dollars. Leaf A predicts 17 thousand dollars.

Step 2: Calculate Leaf B’s prediction. Its response total is $29+40=69$ thousand dollars. There are two responses, so its mean is $69/2=34.5$ thousand dollars. Leaf B predicts 34.5 thousand dollars.

Step 3: Predict for the new observation. The new observation is routed to Leaf B, so it receives Leaf B’s prediction: 34.5 thousand dollars. The new observation’s unknown response is not included in the training mean.

Check. The five-response overall mean would be $(51+69)/5=24$ thousand dollars, which is neither leaf’s mean. Using 24 for both leaves would ignore the given leaf membership and would not minimize squared error separately within each leaf.

Worked Example: Why a Median Is Not the Squared-Error Optimum

A hypothetical terminal leaf contains four training responses of 2, 4, 4, and 10 units. A proposed prediction is 4 units, the median of these responses. Determine the squared-error-optimal prediction and compare the two choices.

Method. Minimize the sum of squared residuals, so calculate the mean and evaluate both candidates. The common mistake to avoid is using the median simply because it is a familiar measure of centre. The median is associated with minimizing absolute deviations, not generally squared deviations.

Step 1: Calculate the mean. The response total is $2+4+4+10=20$ units, and $n=4$. Therefore, $\bar{y}=20/4=5$ units. The squared-error-optimal prediction is 5 units.

Step 2: Calculate squared error at 5. The residuals are $2-5=-3$, $4-5=-1$, $4-5=-1$, and $10-5=5$ units. The squared errors are 9, 1, 1, and 25 units squared, giving a total of $9+1+1+25=36$ units squared.

Step 3: Calculate squared error at 4. The residuals are $-2$, 0, 0, and 6 units. The squared errors are 4, 0, 0, and 36 units squared, giving a total of $4+0+0+36=40$ units squared.

Check. The alternative 4 is one unit below the mean. The identity predicts an increase of $4(4-5)^2=4$ units squared. The observed difference is $40-36=4$. Thus 5, not the median 4, minimizes squared error.

What the Mean Result Does and Does Not Say

The mean result is conditional on the modelling choice: one constant prediction for the observations in a fixed leaf, chosen to minimize their unweighted sum of squared residuals. That precision matters. It does not establish that all observations in the leaf have identical responses, that the mean is a typical individual response, or that the training error is a reliable estimate of prediction error on new data.

The choice of loss function matters. If the objective is absolute error, the median is a minimizing constant (with possible non-uniqueness for an even number of observations). For squared error, the mean is the minimizing constant because deviations from the mean sum to zero and the squared-error identity adds a nonnegative penalty for moving away from it. Do not transfer the mean result to a different objective without checking the relevant minimization problem.

Likewise, the result assumes equal contribution from each observation. If a problem explicitly assigns observation weights, the objective changes to a weighted sum of squared errors, and the appropriate constant is the weighted mean, provided the weights are nonnegative and have a positive total. If the problem does not specify weights, do not invent them.

Key takeaway: For a nonempty terminal leaf with equally weighted training observations, a regression tree’s constant prediction under squared-error loss is the mean response in that leaf. Calculate it from the leaf’s observations only, retain the response’s units, and use the loss function stated in the problem.

Check Your Understanding

For each question, assume equal observation weights and a fixed leaf unless the question states otherwise.

A leaf contains response values 6, 10, and 14. What constant prediction minimizes the sum of squared errors? (Answer: 10)
A leaf contains responses 3, 5, 8, and 12. Calculate its squared-error-optimal prediction. (Answer: 7)
A tree has two leaves. Leaf A contains responses 20, 30, and 50; Leaf B contains 40 and 60. Find the prediction in each leaf, rounding to two decimal places where needed. (Answer: Leaf A: 33.33; Leaf B: 50)
A leaf contains responses 1, 5, 8, and 10. Calculate the sum of squared errors at the optimal prediction, then calculate it at the alternative prediction 8. (Answer: Optimal prediction 6 with squared error 46; at 8, squared error 62)
A new observation is routed to a leaf whose training responses are 4, 6, and 9. What prediction does the leaf give? (Answer: $19/3$, or approximately 6.33)

What You'll Learn