Why Squared-Error Boosting Fits Residuals

Why Residuals Appear in Squared-Error Boosting

In squared-error boosting, each new tree is fitted using information about what the current model has not yet predicted well. For this particular loss, that information is especially simple: it is the residual, the observed response minus the current prediction. The reason is a derivative calculation, not a separate rule to memorize.

The distinction between the derivative and its negative matters. The derivative of squared error points in the direction in which loss increases as a prediction changes. Its negative points toward decreasing loss. Keeping the sign straight explains why the residual—not the prediction minus the observation—is used as the target for the next tree.

Definition: For observation \(i\), let \(y_i\) be its numeric response and \(F_i\) its current prediction. Its residual is \(r_i=y_i-F_i\). Under the half squared-error convention, its loss is \(\ell_i(F_i)=\frac{1}{2}(y_i-F_i)^2\). The negative derivative of this loss with respect to \(F_i\) is the negative gradient for that observation.

The factor \(\frac{1}{2}\) is a convenient convention: it cancels the factor 2 that appears when a square is differentiated. It does not change which predictions minimize squared error. Some summaries use a sum of half squared errors; others use their mean. These choices change the scale of the aggregate loss and its derivative, so it is important to state which one is being used.

Deriving the Negative Gradient

Hold the observed response \(y_i\) fixed and differentiate the per-observation loss with respect to the current prediction \(F_i\). Since the derivative of \(y_i-F_i\) with respect to \(F_i\) is \(-1\), the chain rule gives:

\frac{\partial \ell_i}{\partial F_i} =\frac{\partial}{\partial F_i}\left[\frac{1}{2}(y_i-F_i)^2\right] =\frac{1}{2}\,2(y_i-F_i)(-1) =F_i-y_i.

Taking the negative gives:

-\frac{\partial \ell_i}{\partial F_i}=y_i-F_i=r_i.

This is the central result: the negative gradient of half squared-error loss for an observation is exactly its residual. A regression tree fitted to these negative gradients is therefore fitted to the current residuals. The tree provides a structured set of corrections to the current predictions.

For clarity, suppose the aggregate objective is the mean half squared-error loss, \(J=\frac{1}{n}\sum_{i=1}^{n}\ell_i(F_i)\). Its derivative with respect to one prediction \(F_i\) is \((F_i-y_i)/n\), so its negative derivative is \(r_i/n\). The factor \(1/n\) is common to every observation. Thus, using residuals or residuals divided by \(n\) gives the same target ordering, but the numerical scale differs. When referring to residual targets below, the convention is the negative gradient of the per-observation loss.

Result: With per-observation loss \(\ell_i(F_i)=\frac{1}{2}(y_i-F_i)^2\), the negative gradient is \(y_i-F_i\). If the loss is multiplied by a common positive constant, or averaged over observations, the negative gradient is multiplied by that constant; its direction remains the residual direction.

From Residual Targets to an Additive Update

In a stagewise additive update, the current model is augmented by a fitted tree. Write \(h_m(x_i)\) for the tree's output at observation \(i\), and let \(\nu\) be the multiplier specified for this update. The predictions change according to:

F_m(x_i)=F_{m-1}(x_i)+\nu h_m(x_i).

For squared-error boosting, the tree is fitted using the current residuals \(y_i-F_{m-1}(x_i)\) as targets. Its outputs need not equal each residual exactly: the tree's structure may represent only an approximation to the pattern in them. After the update, a new residual is calculated from the updated prediction. The next tree, if added, is fitted to those new residuals rather than to the original ones.

Calculate current residuals.
For every observation, subtract its current prediction from its observed response: \(r_i=y_i-F_{m-1}(x_i)\).

Fit a tree to the residual targets.
The tree represents a prediction of the negative-gradient pattern from the available predictors.

Update the predictions.
Add the tree's output, multiplied by the stated update multiplier, to each current prediction.

Recalculate residuals and loss.
Use the updated predictions. The residuals generally change, and the loss must be computed using the stated convention.

Worked Example: Derivatives and Residual Targets

A hypothetical pricing analysis has three observations. Their responses are \(y=(10,14,20)\), and the current predictions are \(F=(8,17,18)\). Use per-observation half squared-error loss to calculate each residual, derivative, and negative gradient. Then calculate the mean half squared-error loss.

Step 1: Calculate residuals. Subtract the prediction from the response for each observation:

r_1=10-8=2,\qquad r_2=14-17=-3,\qquad r_3=20-18=2.

Step 2: Differentiate at each prediction. The derivative is \(F_i-y_i=-r_i\), and the negative derivative is \(r_i\). Therefore, the derivatives are \(-2,3,-2\), and the negative gradients are \(2,-3,2\).

Step 3: Calculate the mean loss. The squared residuals are \(2^2=4\), \((-3)^2=9\), and \(2^2=4\). Their sum is \(17\). The mean half squared-error loss is:

J=\frac{1}{3}\left(\frac{4}{2}+\frac{9}{2}+\frac{4}{2}\right) =\frac{17}{6} \approx 2.8333.

Check. Each negative gradient has the same value and sign as its residual, including the negative residual for observation 2, whose current prediction exceeds its response. This example avoids the common mistake of reporting \(F_i-y_i\) as the residual target; that quantity is the derivative, not its negative.

Worked Prediction Updates

A residual-based update has two separate calculations: first obtain the tree outputs by fitting to the residual targets, then add the specified corrections to the current predictions. A small loss after the update is a useful check, but fitting residuals does not by itself guarantee that every possible tree update reduces loss. The fit, tree structure, and size of the update all matter.

Worked Example: Updating Predictions With a Fitted Tree

For the three observations in the preceding example, suppose a fitted regression tree returns outputs \(h=(1,-1,2)\) on those observations. Use the stipulated update multiplier \(\nu=0.5\). Calculate the updated predictions, updated residuals, and mean half squared-error loss. Treat the tree outputs and multiplier as given.

Step 1: Calculate the prediction corrections. Multiply each tree output by \(0.5\): \(0.5(1)=0.5\), \(0.5(-1)=-0.5\), and \(0.5(2)=1\).

Step 2: Update the predictions. Add each correction to the corresponding current prediction:

F_{\text{new}}=(8+0.5,\;17-0.5,\;18+1)=(8.5,\;16.5,\;19).

Step 3: Recalculate residuals. Subtract the updated predictions from the observed responses: \(10-8.5=1.5\), \(14-16.5=-2.5\), and \(20-19=1\). Thus, the new residuals are \((1.5,-2.5,1)\).

Step 4: Calculate the new mean loss. The squared residuals are \(2.25\), \(6.25\), and \(1\), with sum \(9.5\). Hence:

J_{\text{new}}=\frac{1}{3}\left(\frac{2.25+6.25+1}{2}\right) =\frac{9.5}{6} \approx 1.5833.

The initial mean loss was \(17/6\approx2.8333\), so it fell to \(9.5/6\approx1.5833\) for this update. The squared-error reductions can also be checked observation by observation: the squared residual changed from \(4\) to \(2.25\), from \(9\) to \(6.25\), and from \(4\) to \(1\). This example avoids assuming that a tree output is itself the full prediction; it is a correction added to the current prediction.

A Loss-Change Check

The relationship between residuals and loss can be made explicit. If an observation has current residual \(r=y-F\), and its prediction is changed by a correction \(c\), its new residual is \(r-c\). The change in its half squared error is therefore:

\frac{1}{2}(r-c)^2-\frac{1}{2}r^2 =\frac{1}{2}(r^2-2rc+c^2)-\frac{1}{2}r^2 =-rc+\frac{1}{2}c^2.

A correction with the same sign as the residual tends to reduce loss, because the term \(-rc\) is then negative. But the positive term \(c^2/2\) also matters: a correction that is too large can increase the loss. This algebra gives a direct check on an update without claiming that every residual-fitted tree must improve every observation—or even the overall loss.

For a collection of observations with corrections \(c_i\), the change in the sum of half squared errors is \(\sum_i(-r_ic_i+c_i^2/2)\). The change in the mean is that sum divided by \(n\). The sign of the change depends on the balance between alignment with the residuals and the squared size of the corrections.

Worked Example: Sum Loss Versus Mean Loss

A hypothetical model has responses \(y=(8,14)\) and current predictions \(F=(6,17)\). A fitted tree returns outputs \(h=(1,-2)\), and the specified update multiplier is \(\nu=0.5\). Calculate the sum and mean half squared-error losses before and after the update. Explain how using the sum instead of the mean affects the reported values.

Step 1: Calculate initial residuals. \(8-6=2\) and \(14-17=-3\), so \(r=(2,-3)\). The sum of squared residuals is \(2^2+(-3)^2=4+9=13\).

Step 2: Calculate initial losses. The sum of half squared errors is \(13/2=6.5\). With \(n=2\), the mean half squared-error loss is \(6.5/2=3.25\), equivalently \(13/(2\cdot2)=3.25\).

Step 3: Apply the update. The corrections are \(0.5(1)=0.5\) and \(0.5(-2)=-1\). Thus the updated predictions are \((6.5,16)\).

Step 4: Calculate updated residuals and losses. The residuals are \(8-6.5=1.5\) and \(14-16=-2\). The sum of their squares is \(1.5^2+(-2)^2=2.25+4=6.25\). Therefore, the sum half squared-error loss is \(6.25/2=3.125\), and the mean half squared-error loss is \(3.125/2=1.5625\), equivalently \(6.25/(2\cdot2)=1.5625\).

Check and common mistake. The sum loss decreases from \(6.5\) to \(3.125\); the mean loss decreases from \(3.25\) to \(1.5625\). The sum is twice the mean because there are two observations. Reporting \(6.5\) and \(3.125\) as the mean losses would omit division by \(n\). This example avoids mixing the two conventions.

Common Sign and Scale Traps

Reversing the residual sign. The residual is \(y_i-F_i\), and it equals the negative derivative. The derivative itself is \(F_i-y_i\). Fitting the derivative as though it were the negative gradient reverses the intended correction direction.
Forgetting the loss convention. With mean half squared-error loss, divide the sum of squared residuals by \(2n\). With sum half squared-error loss, divide by \(2\), not by \(2n\). State the convention before comparing numerical losses.
Reusing old residuals after an update. Residuals are defined relative to the current predictions. After predictions change, calculate \(y_i-F_{\text{new}}(x_i)\) before describing the new residual targets.
Assuming every update lowers loss. A correction can be misaligned with residuals or too large. Check the updated predictions and loss rather than inferring improvement solely from the fact that a tree was fitted to residuals.

Key takeaway: Under per-observation half squared-error loss, differentiating with respect to the current prediction gives \(F_i-y_i\). Its negative is \(y_i-F_i\), exactly the current residual. Squared-error boosting uses those residuals as targets for the next tree, then updates predictions using the tree's fitted outputs.

Check Your Understanding

Work through each item, keeping the derivative sign and the loss convention explicit.

A response is \(12\) and its current prediction is \(9\). Find the residual, the derivative of half squared-error loss, and its negative gradient. (Answer: Residual \(3\); derivative \(-3\); negative gradient \(3\).)
For two observations, the responses are \((5,11)\) and predictions are \((7,8)\). Find both residual targets and the mean half squared-error loss. (Answer: Residuals \((-2,3)\); mean half squared-error loss \(13/4=3.25\).)
Current predictions are \((4,10,15)\), and fitted tree outputs are \((2,-1,0.5)\). With update multiplier \(0.4\), find the updated predictions. (Answer: \((4.8,9.6,15.2)\).)
Responses are \((10,16)\), current predictions are \((8,18)\), and fitted tree outputs are \((1,-2)\). With multiplier \(0.5\), calculate the updated residuals and mean half squared-error loss before and after the update. (Answer: Updated residuals \((1,-1)\); mean loss decreases from \(2\) to \(0.5\).)
A model has \(n=3\) observations and sum of squared residuals \(24\). State its sum and mean half squared-error losses. Explain which value would be reported if the objective is the mean half squared-error loss. (Answer: Sum loss \(12\); mean loss \(4\); report \(4\) for the stated mean objective.)

What You'll Learn