From Residuals to a Boosted Regression Fit
Gradient boosting builds a prediction model in stages. At each stage, it uses a new tree to improve the current predictions. For squared-error regression, the improvement has a particularly direct interpretation: the tree is fitted to the current residuals, and its leaf values are scaled by a learning rate before they are added to the model.
The arithmetic is simple only if each quantity is kept distinct. The residuals determine both the tree’s split and its leaf values; the learning rate then scales the updates. In particular, the tree is not generally fitted to the original responses again at every stage. This tutorial follows the complete process on small datasets, including multiple iterations.
Initialization and the Iteration
For squared-error loss, use \(L(y,F)=\frac{1}{2}(y-F)^2\). The initial model is a constant chosen to minimize the total loss. If there are \(n\) observations, this constant is the sample mean of the responses. Subsequent updates use the negative loss gradient evaluated at the current predictions. Since \(-\partial L(y_i,F)/\partial F=y_i-F\), that negative gradient is exactly the current residual.
At an iteration, candidate tree structures are compared by how well their leaves represent the pseudo-residuals. For a given leaf, the squared-error-optimal value is their mean. After the tree is fitted, the learning rate scales each leaf value. If \(\nu=1\), the entire leaf update is added; if \(0<\nu<1\), only a fraction is added.
Set every fitted value to the response mean for squared-error loss.
For each observation, subtract its current fitted value from its response.
Choose a partition that represents the pseudo-residuals well, using residual squared error as the criterion.
Use the mean pseudo-residual in each leaf as its value, multiply by \(\nu\), and add the result to the current fitted value.
Why the Leaf Update Is a Mean
For a fixed tree structure, consider one leaf containing a set \(A\) of observations. If the leaf predicts the constant \(c\), its squared-error criterion on the pseudo-residuals is \(\sum_{i\in A}(r_{im}-c)^2\). Differentiate this expression with respect to \(c\), set the derivative equal to zero, and solve:
The second derivative is \(2|A|>0\) for a nonempty leaf, so this critical point minimizes the criterion. Thus the leaf mean is not an extra adjustment chosen by convention: it is the constant that minimizes squared error within that leaf. This result applies to squared-error loss; a different loss can require a different leaf update.
Worked Example: Initialize and Make the First Shrunk Update
A hypothetical insurer models a numerical claim-cost index \(y\) using one ordered predictor \(x\). The four training observations are \((x,y)=(1,10),(2,14),(3,18),(4,22)\). Initialize a squared-error boosting model, fit a stump split after \(x=2\), and update with learning rate \(\nu=0.25\). Use the mean residual in each leaf.
Method and initialization. The response mean is \((10+14+18+22)/4=64/4=16\), so the initial fitted values are \((16,16,16,16)\). The pseudo-residuals are \(10-16=-6\), \(14-16=-2\), \(18-16=2\), and \(22-16=6\). The residual vector is \((-6,-2,2,6)\).
Fit the stump. The left leaf contains the first two observations, so its mean residual is \((-6-2)/2=-4\). The right leaf contains the last two, so its mean residual is \((2+6)/2=4\). As a check on the fit, the residual squared error within the leaves is \([(-6+4)^2+(-2+4)^2]+[(2-4)^2+(6-4)^2]=[4+4]+[4+4]=16\).
Apply shrinkage. The left update is \(0.25(-4)=-1\), and the right update is \(0.25(4)=1\). Add these to the initial prediction of 16. The updated fitted values are \((15,15,17,17)\).
Check the result. The new residuals are \(10-15=-5\), \(14-15=-1\), \(18-17=1\), and \(22-17=5\). Their squared error is \(25+1+1+25=52\), lower than the initial squared error \(36+4+4+36=80\). This is a check on this particular update, not a guarantee that every tree structure or tuning choice will improve performance on new data. The common mistake is to add the full leaf values \((-4,4)\); with \(\nu=0.25\), only one quarter of each value is added.
Choosing a Split From the Current Residuals
A split is judged by the residual squared error remaining after each resulting leaf is assigned its own mean residual. For a candidate split, calculate the mean in each leaf, subtract that mean from each residual in the leaf, square those deviations, and add them. The chosen split has the smallest criterion among the candidates considered. The calculation must use current pseudo-residuals, not the original responses.
Worked Example: Compare Splits and Update Across Two Iterations
Consider four ordered training observations with \(x=1,2,3,4\) and responses \(y=(2,4,6,8)\). Use squared-error loss, regression stumps, and learning rate \(\nu=0.5\). At each stage, consider splits after observations 1, 2, or 3, and choose a split with the smallest residual squared error.
Initialize. The mean response is \((2+4+6+8)/4=20/4=5\), giving \(F_0=(5,5,5,5)\). The initial pseudo-residuals are \((-3,-1,1,3)\).
Iteration 1: compare the candidate splits. For the split after observation 1, the right-leaf residuals \((-1,1,3)\) have mean \(1\), and their squared deviations sum to \(4+0+4=8\); the singleton contributes zero. For the split after observation 2, the leaf means are \((-3-1)/2=-2\) and \((1+3)/2=2\). Each leaf contributes \(2\), so the total is \(4\). For the split after observation 3, the first-leaf mean is \((-3-1+1)/3=-1\), and its squared deviations sum to \(4+0+4=8\); the singleton contributes zero. Therefore the split after observation 2 minimizes the criterion.
The first tree’s leaf values are \(-2\) and \(2\). After shrinkage, the updates are \(0.5(-2)=-1\) and \(0.5(2)=1\). Thus \(F_1=(4,4,6,6)\), and the new pseudo-residuals are \((-2,0,0,2)\).
Iteration 2: compare the candidate splits. For a split after observation 1, the right-leaf residuals \((0,0,2)\) have mean \(2/3\). Their squared deviations sum to \((0-2/3)^2+(0-2/3)^2+(2-2/3)^2=4/9+4/9+16/9=24/9=8/3\). The singleton contributes zero. For a split after observation 2, the two leaf means are \((-2+0)/2=-1\) and \((0+2)/2=1\); the total squared error is \(1+1+1+1=4\). For a split after observation 3, the first-leaf mean is \((-2+0+0)/3=-2/3\), and the squared deviations sum to \((-2+2/3)^2+(2/3)^2+(2/3)^2=16/9+4/9+4/9=8/3\). The last observation is a singleton. The first and third splits tie at \(8/3\); choose the split after observation 1.
The second tree’s leaf values are \(-2\) for \(x=1\) and \(2/3\) for \(x=2,3,4\). The shrunk updates are \(-1\) and \(1/3\). Adding them to \(F_1\) gives \(F_2=(3,13/3,19/3,19/3)\). The new residuals are \((-1,-1/3,-1/3,5/3)\). Their squared error is \(1+1/9+1/9+25/9=36/9=4\), compared with \(8\) after the first iteration and \(20\) at initialization. The common mistake is to overlook the tie between two candidate splits or to use a leaf mean from the previous iteration; here the second tree is fitted to \((-2,0,0,2)\), not to \((-3,-1,1,3)\).
Worked Example: Check the Split Criterion Before Shrinkage
At a later stage of a hypothetical regression fit, four observations have ordered predictor values \(x=1,2,3,4\), responses \(y=(5,6,10,14)\), and current fitted values \(F=(8,7.5,8.5,10.5)\). Fit the best stump among the three possible splits, then update with \(\nu=0.4\).
Calculate the pseudo-residuals. Subtract the fitted values from the responses: \(5-8=-3\), \(6-7.5=-1.5\), \(10-8.5=1.5\), and \(14-10.5=3.5\). Thus \(r=(-3,-1.5,1.5,3.5)\).
Evaluate all splits. After observation 1, the remaining residuals have mean \((-1.5+1.5+3.5)/3=3.5/3=7/6\). Their squared deviations sum to \((-1.5-7/6)^2+(1.5-7/6)^2+(3.5-7/6)^2=64/9+1/9+49/9=114/9=38/3\). After observation 2, the leaf means are \((-3-1.5)/2=-2.25\) and \((1.5+3.5)/2=2.5\). The first leaf contributes \((-3+2.25)^2+(-1.5+2.25)^2=0.5625+0.5625=1.125\); the second contributes \((1.5-2.5)^2+(3.5-2.5)^2=1+1=2\). The total is \(3.125\). After observation 3, the first three residuals have mean \((-3-1.5+1.5)/3=-1\). Their squared deviations sum to \((-3+1)^2+(-1.5+1)^2+(1.5+1)^2=4+0.25+6.25=10.5\); the singleton contributes zero. Therefore the split after observation 2 is best.
Update the fitted values. The leaf means are \(-2.25\) and \(2.5\), so the shrunken updates are \(0.4(-2.25)=-0.9\) and \(0.4(2.5)=1\). The updated fitted values are \((8-0.9,7.5-0.9,8.5+1,10.5+1)=(7.1,6.6,9.5,11.5)\). The resulting residuals are \((-2.1,-0.6,0.5,2.5)\). The common mistake is to miscalculate a squared deviation: for example, \(1.5\) is \(2.5\) above the mean \(-1\), so its squared deviation is \(6.25\), not \(4\).
Reading and Checking a Boosting Fit
A complete hand calculation should make the stage of every quantity clear. At iteration \(m\), the pseudo-residuals are based on \(F_{m-1}\); the tree’s leaf means are calculated from those pseudo-residuals; and the scaled leaf values are added to \(F_{m-1}\) to produce \(F_m\). Keeping these steps in order prevents a common error: repeatedly fitting a tree to the original response values rather than to the current negative gradients.
For squared-error boosting, the negative gradient and ordinary residual coincide. That makes residual fitting a convenient way to calculate the next tree, but the learning rate remains part of the update. The leaf means are the unshrunk tree values; the actual model change is \(\nu\) times those values. When \(\nu<1\), several smaller stages may be needed to make changes that a full step would make at once.
- Check that the initial constant is the response mean.
- Recalculate each residual from the current fitted value before fitting the next tree.
- For every candidate split, use the mean residual within each leaf and compute its squared-deviation total.
- Apply the learning rate to the leaf value before adding it to the fitted value.
- After updating, subtract the new fitted values from the responses to verify the next residual vector.
These calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. The main exam trap is mixing quantities from different stages: a plausible leaf mean or split score can still be wrong if it was calculated from stale residuals or if shrinkage was applied in the wrong place.
Check Your Understanding
For each problem, keep the current fitted values, pseudo-residuals, leaf values, and shrunken updates distinct.
- For responses \(y=(3,7,8,10)\), find the squared-error initial constant and the initial pseudo-residuals. (Answer: \(F_0=7\); residuals \((-4,0,1,3)\).)
- Current residuals in a leaf are \((-2,1,4)\). Find the squared-error-optimal leaf value and its residual squared error. (Answer: leaf value \(1\); squared error \(9+0+9=18\).)
- A stump has leaf values \(-3\) and \(2\), and the current fitted value for one observation in the second leaf is \(11\). With \(\nu=0.2\), find its updated fitted value. (Answer: \(11+0.2(2)=11.4\).)
- For residuals \((-2,0,0,2)\), compare splitting after the first observation with splitting after the second. Give each residual squared error and identify the better split. (Answer: after the first, \(8/3\); after the second, \(4\); the first split is better.)
- Current fitted values are \((4,5,7)\), responses are \((3,7,8)\), and a fitted tree puts observations 1 and 2 in one leaf and observation 3 in the other. Find the pseudo-residuals, both leaf values, and the updated fitted values for \(\nu=0.5\). (Answer: residuals \((-1,2,1)\); leaf values \(0.5\) and \(1\); updated fitted values \((4.25,5.25,7.5)\).)