From the boosting rule to a numerical fit
Gradient boosting builds a prediction in stages: each new tree is fitted to what the current model has not yet explained. For squared-error regression, those quantities are ordinary residuals. Working through the stages by hand makes the roles of the initial prediction, tree splits, leaf values, and learning rate explicit; a short R implementation can then check that the calculations agree.
This tutorial uses small regression trees with one split, called stumps. A stump divides observations into two leaves. The same calculations apply to a tree with more leaves, but a stump keeps the split and leaf arithmetic visible. The response is numeric, and the training objective is the sum of squared errors.
The tree split is selected to make the residuals within its leaves as similar as possible: for each candidate split, calculate the sum of squared deviations from the mean residual in each resulting leaf. The candidate with the smallest total is preferred, subject to any specified tree restrictions. In the worked calculations below, the candidate splits are between ordered predictor values, and both leaves must contain at least one observation.
Why the leaf prediction is the residual mean
A leaf assigns one constant \(c\) to every observation in that leaf. If the residuals in it are \(r_1,\ldots,r_q\), the best constant minimizes their squared error. Differentiating the objective gives the minimizer, and the identity below also shows directly why any other constant has no smaller error.
To verify the identity, write \(r_j-c=(r_j-\bar r)+(\bar r-c)\), square, and sum. The cross-term is \(2(\bar r-c)\sum_j(r_j-\bar r)=0\), because deviations from their mean sum to zero. Since \(q(c-\bar r)^2\geq 0\), the smallest squared error occurs at \(c=\bar r\). This is why the fitted value in each residual-tree leaf is its mean residual.
The update is a shrunken step toward that leaf value. With \(\nu<1\), the tree's contribution is smaller than its fitted residual mean; the update does not replace the old prediction with the tree prediction. This stagewise calculation is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Worked Example: Choosing a stump split
A hypothetical pricing analyst has six training records, with ordered predictor values \(x=1,\ldots,6\) and response values \(y=(-4,-3,-2,2,3,4)\). Fit the initial squared-error model and choose the best one-split tree using residual means in the leaves. Show the candidate split errors.
Step 1: Initialize. The response sum is \(-4-3-2+2+3+4=0\), so \(\bar y=0\) and \(F_0(x)=0\). The initial residuals are therefore \((-4,-3,-2,2,3,4)\).
Step 2: Calculate candidate errors. For a split after \(x=k\), calculate each leaf's residual mean and add its within-leaf squared deviations. For the split after \(x=2\), the left mean is \((-4-3)/2=-3.5\), giving error \((-4+3.5)^2+(-3+3.5)^2=0.5\). The right mean is \((-2+2+3+4)/4=1.75\), giving error \((-2-1.75)^2+(2-1.75)^2+(3-1.75)^2+(4-1.75)^2=20.75\). The total is \(0.5+20.75=21.25\).
Repeating this calculation for all five candidate splits gives:
| Split after | Left-leaf mean | Right-leaf mean | Total within-leaf squared error |
|---|---|---|---|
| \(x=1\) | \(-4\) | \(0.8\) | \(38.8\) |
| \(x=2\) | \(-3.5\) | \(1.75\) | \(21.25\) |
| \(x=3\) | \(-3\) | \(3\) | \(4\) |
| \(x=4\) | \(-1.75\) | \(3.5\) | \(21.25\) |
| \(x=5\) | \(-0.8\) | \(4\) | \(38.8\) |
For example, after \(x=3\), the left squared error is \((-4+3)^2+(-3+3)^2+(-2+3)^2=2\), and the right squared error is \((2-3)^2+(3-3)^2+(4-3)^2=2\). The total \(4\) is the smallest candidate error, so the selected stump splits after \(x=3\), with leaf predictions \(-3\) and \(3\).
Check. The candidate errors are nonnegative, the best split has the smallest total, and its two fitted leaf values are the corresponding residual means. The stump has not yet been added to the model; its contribution depends on the learning rate.
Updating fitted values stage by stage
The next example uses the selected stump repeatedly to make the successive predictions easy to compare. At every stage, residuals are recalculated from the current model. Reusing the original responses as residuals at every stage would repeat the first update instead of correcting the current model.
Worked Example: Three shrunken stump updates
Use the six-record data from the preceding example. At each stage, use a stump splitting after \(x=3\), and use learning rate \(\nu=0.5\). Calculate the training predictions through stage 3 and check the final residuals.
Stage 0. The initial prediction for every record is \(F_0=0\). Thus the residuals are \((-4,-3,-2,2,3,4)\). The two leaf means are \((-4-3-2)/3=-3\) and \((2+3+4)/3=3\). The stage-1 contributions are \(0.5(-3)=-1.5\) on the left and \(0.5(3)=1.5\) on the right. Therefore \(F_1=(-1.5,-1.5,-1.5,1.5,1.5,1.5)\).
Stage 1 residuals. Subtract the new predictions from the responses: \((-4+1.5,-3+1.5,-2+1.5,2-1.5,3-1.5,4-1.5)=(-2.5,-1.5,-0.5,0.5,1.5,2.5)\). The left mean is \((-2.5-1.5-0.5)/3=-1.5\), and the right mean is \((0.5+1.5+2.5)/3=1.5\). After multiplying by \(0.5\), the contributions are \(-0.75\) and \(0.75\). Hence \(F_2=(-2.25,-2.25,-2.25,2.25,2.25,2.25)\).
Stage 2 residuals. The responses minus \(F_2\) are \((-1.75,-0.75,0.25,-0.25,0.75,1.75)\). Their left mean is \((-1.75-0.75+0.25)/3=-0.75\); their right mean is \((-0.25+0.75+1.75)/3=0.75\). The shrunken contributions are \(-0.375\) and \(0.375\), giving \(F_3=(-2.625,-2.625,-2.625,2.625,2.625,2.625)\).
Check. The final residuals are \((-1.375,-0.375,0.625,-0.625,0.375,1.375)\). Their sums within the left and right leaves are both \(-1.125\) and \(1.125\), respectively. The fitted residual means at stage 3 are therefore \(-0.375\) and \(0.375\), exactly the contributions added at that stage before shrinkage when multiplied by \(0.5\). The response values are not yet perfectly fitted: shrinkage deliberately makes each step smaller.
Reproducing the updates in base R
A compact base R loop can calculate residuals, evaluate every possible stump split, select the lowest-error split, and update predictions. This implementation deliberately uses only one split per tree. Its printed values can be compared directly with the hand calculations. Ties are resolved by which.min in favor of the first minimizing split; in this example the best split is unique at every stage.
Worked Example: Fitting and checking the same model in R
Fit three stages to the six records, using \(\nu=0.5\). At each stage, find the split that minimizes the sum of within-leaf squared residuals, then print the split, leaf values, and training predictions.
R implementation:
x <- 1:6
y <- c(-4, -3, -2, 2, 3, 4)
nu <- 0.5
F <- rep(mean(y), length(y))
for (m in 1:3) {
r <- y - F
split.error <- sapply(1:5, function(k) {
sum((r[1:k] - mean(r[1:k]))^2) +
sum((r[(k + 1):6] - mean(r[(k + 1):6]))^2)
})
k <- which.min(split.error)
left.value <- mean(r[1:k])
right.value <- mean(r[(k + 1):6])
F <- F + nu * ifelse(x <= k, left.value, right.value)
cat("stage", m, "split", k, "leaf values",
left.value, right.value, "predictions", F, "\n")
}
The first iteration evaluates errors \(38.8, 21.25, 4, 21.25, 38.8\), selects \(k=3\), and gives leaf values \(-3\) and \(3\). The next two iterations also select \(k=3\), with leaf values \((-1.5,1.5)\) and then \((-0.75,0.75)\). The printed prediction vectors are, in order, \((-1.5,-1.5,-1.5,1.5,1.5,1.5)\), \((-2.25,-2.25,-2.25,2.25,2.25,2.25)\), and \((-2.625,-2.625,-2.625,2.625,2.625,2.625)\), matching the hand calculations.
Check. The loop initializes \(F\) to the response mean, recomputes \(r=y-F\) inside each iteration, and adds—not replaces—the shrunken tree prediction. These details are necessary for the code to implement the stated stagewise model.
Arithmetic checks and common exam traps
The candidate split calculations above are sums of squared deviations from each leaf's own mean. An arithmetic expression using a different, fixed leaf prediction can have a different value; it must not be reported as the optimal within-leaf error. For clarity, consider the five numbers \((-4,-3,-2,2,3)\). Their mean is \(-0.8\), and the sum of squared deviations from their own mean is \((-4+0.8)^2+(-3+0.8)^2+(-2+0.8)^2+(2+0.8)^2+(3+0.8)^2=38.8\).
If instead the prediction is fixed at \(0.8\), the squared-error sum is \((-4-0.8)^2+(-3-0.8)^2+(-2-0.8)^2+(2-0.8)^2+(3-0.8)^2=51.6\). If it is fixed at \(1.75\), the sum is \((-4-1.75)^2+(-3-1.75)^2+(-2-1.75)^2+(2-1.75)^2+(3-1.75)^2=71.3125\), or \(71.31\) to two decimal places. Both fixed-prediction calculations are valid for those stipulated predictions, but neither is the optimally fitted leaf error: the optimal prediction is \(-0.8\), yielding \(38.8\). Separating the prediction value from the data values prevents a plausible-looking arithmetic result from being mistaken for the fitted leaf criterion.
- Using the wrong residuals. After a stage update, calculate \(r_i=y_i-F_m(x_i)\). Do not continue fitting the initial residuals.
- Forgetting shrinkage. Add \(\nu h_m(x)\), not the full \(h_m(x)\), when \(\nu<1\).
- Comparing unsquared errors. Split selection for squared-error boosting uses sums of squared deviations, not sums of residuals or absolute deviations.
- Mixing up a mean and a fixed prediction. The optimal unconstrained constant leaf value is the residual mean. If a calculation uses another value, label it as fixed and do not call its error the optimal leaf error.
Check Your Understanding
For each problem, track the stated initial prediction, residuals, and learning-rate update carefully.
- A three-record regression sample has responses \(2,4,6\) and initial prediction \(4\) for every record. A stump splits after the second record, and the learning rate is \(0.2\). Find the leaf residual means and the updated predictions. (Answer: leaf means \(-1\) and \(2\); updated predictions \(3.8, 3.8, 4.4\).)
- Current predictions for four records are all \(10\), and the responses are \(6,8,12,14\). A fixed stump puts the first two records in one leaf and the last two in the other. With learning rate \(0.25\), find the next predictions. (Answer: residual means \(-3\) and \(3\); predictions \(9.25, 9.25, 10.75, 10.75\).)
- For ordered responses \(1,3,5,7\), initialize at their mean and fit one stump splitting after the second record. Use learning rate \(0.5\). Find the residual leaf means and predictions after the update. (Answer: initial prediction \(4\); leaf means \(-2\) and \(2\); updated predictions \(3,3,5,5\).)
- Residuals at an iteration are \((-3,-1,1,3)\). Compare the within-leaf errors for splits after the first and second observations, using the residual mean in each leaf. Which split is better? (Answer: split after the first has error \(8\); split after the second has error \(4\), so the second split is better.)
- A fitted model currently predicts \(5\) for each of three observations with responses \(2,5,8\). One leaf contains all three observations, and its fitted residual mean is to be added with learning rate \(0.4\). Find the leaf value, the updated prediction, and the sum of squared residuals after the update. (Answer: residuals \(-3,0,3\); leaf value \(0\); updated predictions remain \(5\); squared-error sum \(18\).)