Why Fit a Tree to Residuals?
In squared-error boosting, each new tree is added to the model already built. The new tree is not asked to predict the original response from scratch. Instead, it is fitted to what the current model has not yet explained: the current residuals. This makes each stage a targeted correction to the existing predictions.
The key claim is about training squared error. If the next tree is fitted to the residuals by least squares, and the available tree fits include the zero fit, then adding that tree cannot increase the squared error on the training observations. The claim does not say that every conceivable tree update improves the fit, or that training improvement guarantees better predictions on new data.
A residual is signed: a positive residual means the current prediction is below the observed response, and a negative residual means it is above. Consequently, the added tree’s predictions can be positive in some regions and negative in others. The update is additive: each new prediction is added to the current one.
The Squared-Error Improvement
At a given stage, the current predictions are fixed. The remaining problem is to find a tree whose predictions approximate the current residuals. For any candidate tree \(h\), the response error after adding it is
Thus, minimizing squared response error after the update is exactly the same as minimizing squared error when fitting the tree to the current residuals. The following derivation makes the improvement condition precise.
Because the new response error is \(r_{im}-h(x_i)\), the updated sum of squares is \(\sum_i [r_{im}-h(x_i)]^2\).
The current sum of squares is \(\sum_i r_{im}^2\). Subtracting it from the updated error gives \(\sum_i h(x_i)^2-2\sum_i r_{im}h(x_i)\).
If the tree is selected to minimize \(\sum_i[r_{im}-h(x_i)]^2\), and the tree class includes \(h(x_i)=0\) for every observation, the chosen fit has error no greater than that zero fit. Therefore the updated training error is no greater than the current training error.
The result is non-increase, not necessarily strict decrease. If the best available tree is the zero fit, or if its predictions do not reduce the residual sum of squares, the error may remain unchanged. Also, the guarantee applies to a tree chosen to fit the current residuals under squared error. An arbitrary tree prediction can make the error larger. This is a statement about the training observations; it does not establish how well the updated model will perform on new observations.
Worked Prediction Updates
Worked Example: One Tree Corrects a Current Model
Suppose a hypothetical insurer groups four policies by a rating variable. The current model predicts claim costs of \([100,100,120,120]\) dollars, and the observed costs are \([104,102,116,118]\) dollars. A candidate tree fitted to the current residuals predicts \([3,3,-3,-3]\) dollars. Calculate the residuals, updated predictions, and change in training squared error.
Method. Compute observed minus current prediction for each policy, add the tree’s prediction to the current prediction, and compare the two sums of squared errors.
The current residuals are \(104-100=4\), \(102-100=2\), \(116-120=-4\), and \(118-120=-2\). The current sum of squared errors is \(4^2+2^2+(-4)^2+(-2)^2=16+4+16+4=40\).
Add the tree predictions to the current predictions: \([100+3,100+3,120-3,120-3]=[103,103,117,117]\) dollars. The new response errors are \(104-103=1\), \(102-103=-1\), \(116-117=-1\), and \(118-117=1\). The updated sum of squared errors is \(1^2+(-1)^2+(-1)^2+1^2=4\).
Check. The squared error falls from \(40\) to \(4\), a reduction of \(36\). The residuals after the update, \([1,-1,-1,1]\), agree with subtracting the tree predictions from the original residuals. A common mistake is to treat \([3,3,-3,-3]\) as the new model predictions; those values are the corrections, not the updated predictions.
Worked Example: Checking an Imperfect Residual Fit
For four hypothetical equipment accounts, the current predictions are \([20,22,24,26]\) thousand dollars and the observed values are \([25,23,22,22]\) thousand dollars. A fitted tree predicts \([3,2,-1,-2]\) thousand dollars. Calculate the new predictions and verify the change in squared error.
Method. First calculate the current residuals. Then apply the tree additively and compare the residual-fitting error before and after the update.
The current residuals are \([25-20,23-22,22-24,22-26]=[5,1,-2,-4]\). The current sum of squared errors is \(5^2+1^2+(-2)^2+(-4)^2=25+1+4+16=46\).
The updated predictions are \([20+3,22+2,24-1,26-2]=[23,24,23,24]\) thousand dollars. Subtracting these from the observed values gives new errors \([2,-1,-1,-2]\), so the updated sum of squared errors is \(2^2+(-1)^2+(-1)^2+(-2)^2=4+1+1+4=10\).
The algebraic check gives the same result. The sum of squared tree predictions is \(3^2+2^2+(-1)^2+(-2)^2=18\). The sum of residual-times-tree-prediction products is \(5(3)+1(2)+(-2)(-1)+(-4)(-2)=15+2+2+8=27\). Therefore the change in error is \(18-2(27)=-36\), and \(46-36=10\).
Check. The updated error agrees with the direct calculation. The common mistake avoided here is to assume that the tree must reproduce every residual exactly; an imperfect residual fit can still lower the training error.
Worked Example: Recompute Residuals Before the Next Stage
A hypothetical portfolio has observed claim amounts \([54,48,52,46]\) dollars and a current prediction of \(50\) dollars for each observation. The first tree predicts \([3,-1,1,-3]\) dollars. A second tree, fitted after that update, predicts \([1,-1,1,-1]\) dollars. Find the predictions and residuals after each stage, and check the training squared errors.
Method. At each stage, add that stage’s tree to the current predictions. Calculate the next residuals from the updated predictions rather than reusing the original residuals.
Initially, the residuals are \([54-50,48-50,52-50,46-50]=[4,-2,2,-4]\). Their squared sum is \(4^2+(-2)^2+2^2+(-4)^2=16+4+4+16=40\).
After the first tree, the predictions are \([50+3,50-1,50+1,50-3]=[53,49,51,47]\). The new residuals are \([54-53,48-49,52-51,46-47]=[1,-1,1,-1]\). Their squared sum is \(1+1+1+1=4\).
The second tree is fitted to these updated residuals. Adding its predictions gives \([53+1,49-1,51+1,47-1]=[54,48,52,46]\). The final residuals are \([0,0,0,0]\), and the final squared error is \(0\).
Check. The errors decrease from \(40\) to \(4\) to \(0\). Reusing the initial residuals for the second stage would instead apply corrections intended for a model that no longer exists and would not produce the displayed final predictions.
Conditions and Common Traps
Residual fitting gives a clear stagewise objective, but the improvement statement has specific conditions. The tree must be evaluated on the same training observations used to calculate the current residuals, the fitting criterion must be squared error, and the selected tree must minimize that criterion over a class that includes a zero fit. If a procedure restricts the candidate predictions or uses a tree chosen for another purpose, its update need not improve training squared error.
- Replacing instead of adding. The tree output is a correction to the current prediction. The new prediction is \(F_{m-1}(x_i)+h_m(x_i)\), not \(h_m(x_i)\) alone.
- Using stale residuals. After an update, recompute \(y_i-F_m(x_i)\). The next tree targets these new residuals, not the original ones.
- Assuming any update helps. The non-increase result follows from least-squares fitting with the zero fit available. A poorly chosen or arbitrary correction can increase error.
- Confusing training fit with predictive performance. The algebra compares squared errors on the training observations. It does not guarantee an improvement on unseen data.
Residual fitting is useful because it converts an additive modeling problem into a sequence of correction problems. At each stage, the current model determines the errors, and the next tree is aimed at those errors. This is the squared-error mechanism behind stagewise improvement and is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, track signed residuals, add the tree predictions to the current model, and verify squared-error changes where requested.
- A current model predicts \([30,30,34]\) dollars and the observed values are \([34,27,32]\). A fitted tree predicts \([2,-2,-1]\). Find the updated predictions and the old and new sums of squared errors. (Answer: updated predictions \([32,28,33]\); old SSE \(29\), new SSE \(6\).)
- Current residuals are \([6,2,-2,-6]\), and a candidate tree predicts \([4,1,-1,-4]\). What is the squared error before and after a unit update? (Answer: before \(80\), after \(10\).)
- Current predictions are \([10,12,14,16]\), and observed values are \([13,10,12,12]\). A tree predicts \([2,-1,-1,-2]\). Find the updated predictions and verify the new squared error. (Answer: updated predictions \([12,11,13,14]\); new SSE \(7\), down from \(33\).)
- Current residuals are \([2,-1]\), but an arbitrary proposed tree predicts \([1,2]\). Compare the current squared error with the error after adding the tree. Does this update improve the fit? (Answer: current SSE \(5\); updated residuals \([1,-3]\), updated SSE \(10\); no, this arbitrary update worsens the fit.)